Copyright ⓒ 2011, Juniper Networks, Inc. 1
CREATING INNOVATIVE
EMBEDDED APPLICATIONS IN
THE NETWORK WITH
THE JUNOS SDK
This work is an excerpt from the
forthcoming book “Network
Embedded Management and
Applications” expected to be
published in 2011 by Springer Press.
2 Copyright ⓒ 2011, Juniper Networks, Inc.
Creating Innovative Embedded Applications in the Network with the Junos SDK
Evolving a System into a Platform
Juniper Networks, Inc. values innovation and realizes the power of the software that runs the network. The company has always

taken pride in hardware and silicon innovation, building equipment that pushes the performance envelope of the industry. Its

chief software asset, on the other hand, is widely understood to be the Junos software. Junos software is a portfolio in its own

right: the Junos Operating System (OS), which runs the network elements; Junos Space, which runs across the network elements;

and Junos Pulse, which runs on the network endpoints.
The Junos OS is known as a system that runs reliably on top of any supported hardware, while having only one release train

and a consistent operational feel on any device. While customers value these traits, many do not know that since 2007 Juniper

Networks has opened this system to innovation by creating the Junos Software Development Kit (SDK). The Junos SDK underlines

the company’s emphasis on software and commitment to innovation through network programmability.
Network programmability itself, however, is not the goal, but the means towards greater innovation and an evolution of
the network. In addition to creating SDKs, Juniper Networks is evolving its software systems to serve as platforms that
will run network applications. The Junos SDK enables ease of development and reuse of components for collaboration, while the

underlying Junos OS provides security, robustness, and resiliency to failure, and a widespread platform for application

deployment.
Here we look at the Junos SDK and the kinds of applications that it enables. Components and applications can be built,

packaged, deployed, and finally run natively right on the devices like other native software processes. The Junos SDK provides C

and C++ APIs with POSIX compliance, a familiar environment to those with UNIX programming experience. Standard APIs such as

libc and STL are available, along with many other common libraries. The more remarkable parts are the Juniper Networks-specific

APIs pulled from the Junos OS that allow dynamic control of the device’s functions. These APIs evolve, but with compatibility in

mind: the Junos SDK’s tools, APIs, and other elements release regularly in step with the Junos OS. Applications made with the

Junos SDK benefit from the platform portability of the Junos OS. They may run on any device running the standard Junos OS,

and most Juniper Networks hardware products are qualified for such support.
Applications in the Junos Architecture
Junos SDK-built applications integrate with the Junos OS to create innovative solutions in several architectural areas of the device.

Where can you deploy an application, and how can you determine where it is best suited?
We present the Junos OS architecture at a high level and in detail where it pertains to building applications. We continue by

explaining how the architecture fits over the hardware, and what types of applications are best suited to the environments we

describe. The figure at the end of this section summarizes the main conceptual separations.
Partitions of the Junos Architecture and Hardware
The Junos OS is a single network operating system integrating routing, switching, and security. Most Juniper Networks hardware

platforms run the Junos OS, and most of these platforms support Junos SDK-built applications. The network operating system

design has a fundamental logical division into three elements: the control plane, the data plane, and the services plane. Here we

discuss them within the context of Junos and bring out points relevant to applications.
The Control Plane
The control plane’s primary role is to manage and control the behavior of the device, including the other two planes. It must

have a global view of the device hardware and software. It also manages the user interface. The control plane in a Junos OS-

based device runs on hardware called a Routing Engine (RE).
Many Juniper Networks products have REs that are hot-swappable physical modules in the chassis. There is often a chassis

option for a redundant RE backing up the master, as well. Later we shall discuss how the Junos SDK facilitates taking advantage

of this redundant RE in your applications. Whether the RE is a physical module or is integrated into a smaller device, a master RE

must always be present, as it is the primary location of the control plane software that controls the rest of the device.
The control plane’s key element is the Junos OS software. The basis of the Junos kernel comes from the FreeBSD UNIX operating

system, an open-source software system. This mature, general-purpose system provides many of the essential basic functions of

an operating system, such as the scheduling of resources. To transform it into a network operating system, Juniper Networks

engineers have extensively modified and hardened it for the specialized requirements of networking. The
Copyright ⓒ 2011, Juniper Networks, Inc. 3
Creating Innovative Embedded Applications in the Network with the Junos SDK
task of managing the device is shared among the Junos kernel, many Junos daemons, and some ephemeral utility-style tools

launched on demand. The daemons and tools that come bundled with the Junos OS are considered part of the Junos OS

platform. Junos SDK-built applications for the control plane resemble these daemons and tools. In fact, Juniper Networks employs

the Junos SDK APIs internally when building these applications, which then become part of the common platform and Junos OS

software bundle.
The Junos SDK provides the Routing Engine SDK (RE SDK) APIs and tools to build applications to extend the control plane

software on the RE. Because an RE is always present in any device, RE SDK-based applications are always deployable without the

addition of any extra hardware or software.
On the control plane, the Junos kernel and many Junos daemons expose APIs so that other processes can programmatically

manipulate their states and make use of their services in a dynamic way. The daemons that control the Junos user interface also

allow for programmatic and seamless extensibility of the user interface. New applications added to a device can thus be

configured and administered in the same ways as the Junos OS platform applications.
The Data Plane
The data plane spans many aspects of a device’s chassis and its modules. It is collectively referred to and abstracted as the

Packet Forwarding Engine (PFE) and is comprised of ASIC-based hardware and Junos OS microcode that performs packet

processing that is generally stateless. Aiming to perform at fast wire speeds and within its hardware resource limits, the PFE

generally delegates stateful packet processing to the services plane.
The data plane’s chief role is forwarding traffic according to the forwarding table, which is primarily formed through the routing

control service on the control plane. The data plane’s main extended abilities include switching, filtering, rate limiting, shaping,

and other quality-of-service (QoS) functions. These functions are originally controlled in software on the control plane, and much

of that software exposes APIs.
Today, Junos SDK applications do not run in the data plane, but an application running on the control plane or the services

plane can certainly influence packet processing in the data plane by using APIs. Fortunately for application developers, the details

of data plane hardware are abstracted out in APIs that are common across hardware platforms. Some APIs enable use of

features unique to specific hardware platforms.
The Services Plane
The services plane can be thought of as an extension to the data plane to perform stateful services and any services non-native

to the PFE. A classic example of a Junos service application is the stateful firewall.
In traditional forwarding devices, the data plane usually handles most of the packet processing, and the services plane runs on

optionally installable and hot-swappable hardware, which we generically name services modules. These contain one or more

service processing engines, which connect to the PFE at speeds up to 10 Gbps (circa 2010 hardware). The services plane includes

all services modules and engines in a chassis, and a given service application can be deployed on more than one engine or

module.
In security- and service-oriented devices, the services plane is the primary packet processor, and the data plane merely connects

a chassis mostly full of services modules. While these devices can perform forwarding, they are intended to service traffic in

stateful ways implemented in software running on the service engines across the services modules. This focus contrasts with the

devices mentioned above that are purpose built to forward packets in hardware, but the services modules are similar, and the

Junos SDK supports a common model for service application development in either case.
Each service engine runs a Junos OS kernel with real-time processing capabilities. The kernel is essentially the same as the one

present on an RE, but far fewer and different daemons run, since the service engine is geared towards servicing tasks instead of

the general device management done on the RE. Also, the hardware resources differ greatly from those on the RE. Each service

engine consists of a multiprocessing, multithreaded CPU and more memory than the typical RE, so that the software-based

services can quickly process packets in parallel and maintain a large amount of state. The Services SDK encompasses the APIs

and tools to build applications to run on the service engines.
4 Copyright ⓒ 2011, Juniper Networks, Inc.
Creating Innovative Embedded Applications in the Network with the Junos SDK
The Junos kernel on a services engine further logically divides packet handling for applications in two sub-planes. Its services

plane (data plane extension) is for fast customized transit traffic processing. Its control plane is for traffic termination with the

kernel’s IP stack.
The control side here is used to send or terminate traffic, as can be done on the RE or any usual host system. The control sub-

plane inside a service engine can be used to run a control-plane-style application that would normally run on the RE. Such

components frequently implement a server or signaling to communicate outside the device or simply to other components on the

RE. Co-location of such a control-plane-style component with a service-plane-style component benefits many applications because

the two components can share memory and information more easily and with greater performance.
Applications in the services sub-plane of a service engine can take on two roles involving inline packet processing: transforming

and monitoring. Transforming applications have access to any traffic flowing through the PFE that is selected for servicing at the

given service engine. They can modify, drop, hold, and create entire IP packets. Sending and receiving this service traffic by using

Services SDK packet-processing APIs happens in parallel and much more quickly than it would in a program using traditional

sockets. Monitoring applications work similarly, but the packets they receive are duplicates of some traffic originally transiting the

PFE. The original traffic is not impacted or serviced in this case, hence the term “monitoring.” A hybrid application can

differentiate and deal with both original and duplicated packets. Another kind of hybrid application, namely a gateway, may

combine a server or signaling control plane component with a transforming or monitoring services plane component. Many

familiar service applications, such as IPsec, work with both styles of components.
Integrated Hardware
An application built with the Junos SDK can run on either a routing engine (RE) or a services module. On some of the smaller

Juniper Networks devices, physical modules do not plug in to a chassis. Rather, a single enclosure contains the hardware

necessary to run a built-in control, data, and services plane. Nonetheless, applications still are supported in the control and

services planes, and we continue to use the routing engine and services module terminology.
Traffic Types
As applications take on different roles working in either the control or services plane, they deal with two different traffic types.
Control traffic is dealt with on the control plane. This traffic is either entirely internal to the device (e.g., inter-process

communication), or it is traffic that is destined to or sourced from an address of the device. Most addresses configured on the

device belong to the master RE. For example, all addresses configured on network interfaces for I/O ports and the loopback

interface pertain to the RE, so control traffic destined to those addresses is forwarded to the master RE and handled there. The

Junos OS also allows the configuration of addresses on interfaces representing a service engine. Traffic destined to those

addresses is forwarded to and handled on the given service engine’s control sub-plane.
Data traffic flows through the data plane. That which is related to any Junos SDK-built application is the data traffic selected for

servicing or monitoring. This is the traffic seen in the services plane handled by transforming or monitoring applications. Data

traffic would naturally flow through the PFE as transit traffic, but by special mechanisms it can be selected for steering through

the service engine’s services plane. Upon exit, it is rerouted and filtered in the PFE as if it were entering the device from any I/O

interface.
Copyright ⓒ 2011, Juniper Networks, Inc. 5
Creating Innovative Embedded Applications in the Network with the Junos SDK
Figure 1. Architecture summary and traffic paths
Working with Junos Features through the SDK APIs
The APIs of the Junos SDK allow you to make use of the features provided by the Junos OS platform software. Having

applications co-resident on the device naturally creates savings, but combining the existing value of the application with the ability

to utilize Junos OS features allows unprecedented customization of a carrier-class network operating system. The main features

covered in this section show you part of a growing body of platform functions that can be built into your application with the

APIs provided.
Familiar Basics
Given that the basis of the Junos kernel comes from the FreeBSD UNIX operating system, most C and C++ UNIX developers are

familiar with the API basics in the Junos SDK: the POSIX APIs, POSIX threads, the C standard library (libc), the C++ standard

template library (STL), socket programming, and syslog. Developers using the Junos SDK can immediately take advantage of this

knowledge, or even reuse existing code developed to these common standards. Third-party libraries of custom utilities that are

not already packaged in the Junos SDK are generally quick to port and use during application development as well.
One different but basic concept present in the Junos OS for which the Junos SDK provides APIs is that of routing instances.

These are analogous to virtual routers. Communication using sockets is enhanced by optional tying of the socket to a routing

instance. Routing instances normally are configured by the operator, but an internal instance also is reserved for all Junos SDK-

built applications to use for their own IP socket-based inter-process communication (IPC). The Junos SDK provides an IPC library

of APIs that uses these features for internal control traffic. It facilitates IPC on the RE and service engines, or between them,

without the need for IP addressing. Because this library automatically uses the internal routing instance reserved for new

applications, using it means that these applications cannot interfere with the Junos OS platform software’s IPC and other traffic in

the device. Applications also benefit from the inherent security of the internal routing instance: no connections from outside the

machine can be made to these sockets.
Working with Interfaces
The Junos OS primarily manages the chassis hardware with configured network interfaces. Some interfaces are internal as well.

Junos SDK APIs provide access to interface naming and properties, which are considered on four levels: device, logical, protocol

family, and addressing. Applications can register to be notified of interface state changes at each of these levels. For example, a

device interface could go down when the cable to the interface’s port is disconnected, and an event would be generated to the

registered applications. While the RE has a global view of all the interfaces in the device, the Services SDK applications are limited

to seeing only the information for the interfaces local to the service engine on which they run.
Routing Engines – Control PlanePacket Forwarding Engine – Data PlaneService Engine – Service PlaneServices(data)

planeServiceapplicationControl PlaneI/OModulesI/OModulesApplication(s)User interfaceextensionsIngressTra cEgressTra cLocal

control tra cInternal control tra cNon-serviced(transit) datatra cLocal controltra c addressedto service moduleServiced (transit) data

tra cBuilt with theRE SDKBuilt with theRE SDKBuilt with theServices SDK
6 Copyright ⓒ 2011, Juniper Networks, Inc.
Creating Innovative Embedded Applications in the Network with the Junos SDK
The interfaces that manage service engines can also be paired up in a redundant configuration. In such a mode, the redundancy

state and properties are entirely retrievable, and applications can listen for state changes such as switchovers.
In addition, all of the per-interface traffic and error statistics for local and transit traffic typically are accessed through the Junos

UI. RE SDK APIs, however, allow for an application to retrieve these interface statistics programmatically.
Manipulating the Data Plane Functions
Routes and filters traditionally are static and are configured by an operator. Dynamic route manipulation has been possible only

through the use of routing protocols. With the advent of the Junos SDK, routes and filters also can be manipulated quickly by

applications using Junos SDK APIs.This capability allows you to wield the data plane’s line-rate packet processing and scaling of

forwarding and filtering as you wish. The libraries responsible for this functionality work with the RE and Services SDKs, but both

routes and filters are managed by Junos OS platform daemons on the RE and eventually distilled into a forwarding table and set

of applied filters. These entities are sent to the packet forwarding engine for processing in hardware.
Properties of the forwarding tables and the table entries themselves are retrievable for any routing instance via the route-related

APIs. IP route manipulation in any table is also possible. One can programmatically create and delete routes with parameters

equivalent to those of a configured static route.
The internal process of route management in the Junos OS is complex, given the sophisticated nature of the supported routing

protocols, routing instances, and routing policy. Handling of these items is greatly simplified with Junos SDK APIs. These

programmatic abilities, combined with the interface information described in the previous section, allow you to develop a dynamic

routing protocol that is decoupled from the rest of the system.
Stateless firewall filters in the Juniper Networks devices can match packets based on layer 2 through layer 4 protocol header

fields. You can take an array of actions on packets that match such filter conditions. Filtering actions include accepting,

discarding, redirecting, sampling (for monitoring applications), statistics counting, logging, rate limiting, and changing QoS

parameters.
Filters and rate limiters are created to be applied on ingress and egress of network interfaces. Junos SDK APIs allow you to

manipulate, apply, and remove such filters and rate limiters programmatically. Furthermore, for the filters that count packets, the

statistics are equally retrievable with APIs.
Seamless User Interface Integration
Applications on Junos can be administered by several user interfaces, such as the CLI, SNMP, NETCONF, and the Junos OS to

Junos Space Device Management Interface. All interfaces but SNMP function in either an operational or a configuration mode.

Commands issued in the operational mode upgrade software, trigger events, or most often show status and statistics to allow

inspection of the system’s operation. Commands issued in the configuration mode modify the configuration database, which is

persistent across reboots. The syntax for the user interface commands follows a predefined hierarchical schema that can be

extended with the addition of RE-based applications packages.
The RE SDK provides a simple language to facilitate extending the schema for new operational commands. It allows defining the

syntax for commands and registering a new application as the handler. When the application receives a command, it may want to

return some resultant data. This return is facilitated by another new language used to define the structure and display format of

an XML response.
You can extend the schema for the hierarchy of object nodes in the configuration database. This capability allows the operator to

configure an RE SDK-built application with new syntax through any one of the Junos user interfaces. The same language used to

create new operational commands allows defining new configuration syntax for objects. It can also designate points within the

configuration database that, if present, require an application to be run, or if changed, require an application to be notified.
Through Junos SDK APIs, applications have read-only access to the configuration database. These APIs provide easy ways to

detect specific configuration additions, changes, or deletions, so applications can modify their behavior accordingly.
Write access to the configuration database and the ability to call configuration commands are not achieved through APIs. These

capabilities are left under the operator’s control, but with Junos scripting, the operator can run operational scripts to execute

commands and configuration changes. When an operator grants such a script executable permissions, it becomes executable

from an RE SDK-built application as well. Thus an application can effectively manipulate any aspect of the Junos system that is

exposed through the Junos user interface.
Copyright ⓒ 2011, Juniper Networks, Inc. 7
Creating Innovative Embedded Applications in the Network with the Junos SDK
Using Other Junos OS Features on the RE
You can develop innovative applications through use of functionality and features implemented in other components. On the

Junos OS platform, these reusable components could be the Junos daemons themselves, other Junos SDK-built daemons, and

even other Junos SDK-built libraries. Of course, the Junos SDK-built components need not all come from any one source: they

may come from multiple third parties creating software with the Junos SDK.
Here we give a few more examples of other features built into a wide array of applications shipped in the Junos OS platform.
SNMP
You can integrate with the Junos SNMP framework just as seamlessly as with the other user interfaces. Using the publically

available Net-SNMP APIs provided in the RE SDK, applications can register themselves as SNMP sub-agents. As such, they can

register new MIB objects and accordingly handle requests for information retrieval as these are directed to the application. They

can send SNMP traps as well.
AAA
The Junos authentication, authorization, and accounting (AAA) infrastructure has more features that applications can use through

the Junos SDK APIs. For instance, the operator may configure an LDAP or RADIUS server connection for the Junos OS to use for

AAA purposes. This connection is reusable through APIs that communicate through the proxy in the Junos OS to automatically

send authentication requests or accounting information to the configured server. This example shows how an application can

reuse functionality from another component rather than implementing a feature from scratch.
Packet Processing, Data Traffic Distribution, and Concurrency Control
One of the most important features of the Services SDK is that it facilitates the setup and processing of data traffic. Multiple

packet-polling loops, each running on a real-time thread, are dedicated to processing the data traffic in the services plane. The

setup of this feature is quite flexible and controlled partially by the operator through the configuration and partially by the

developer specifying the number of real-time threads. More than twenty real-time packet-processing threads can be set up per

service engine. The configuration parameters and programmatic decisions control the amount of CPU resources dedicated to

packet processing and the degree of parallelism versus speed of processing. The packets that are sent and received using the

Services SDK APIs are delivered to the application as quickly as possible using zero-copy I/O from the kernel to the queues

polled by the user-space application.
Since there are many real-time threads, each with its own queue to poll for packets, the system needs to decide how to distribute

received packets among the many queues. This configurable property can be set one of two ways: round robin, or flow affinity.

The round robin method distributes the packets evenly, cycling through the queues and back again continuously. The flow

affinity method ensures that packets of the same flow, or actually the same 3- or 5-tuple, always are distributed to the same

queue and hence to the same thread. The tuple defined for a flow usually consists of the protocol, along with source and

destination IP addresses, and optionally source and destination ports.
Real-time threads can be created with API extensions added to the normal POSIX thread library, but other Services SDK APIs

simplify this task even further to create and start the threads along with matching packet queues. Each real-time thread runs

without preemption on what we call a virtual CPU because the software thread is exclusively bound to a single hardware thread

of the underlying multi-processing CPU. While these threads are necessary for the packet loops, the Services SDK also provides

APIs allowing the creation of threads for generic real-time processing just as easily.
Although the POSIX thread-locking structures are available, real-time threads benefit more from the use of spinlocks and atomic

operations. The Services SDK provides simple APIs for both, optimized for the service engine environment.
Once an application holds a packet, it may use a comprehensive suite of APIs to access or change its metadata or actual content.

The metadata, for example, contains information such as the ingress interface and modifiable parameters such as the interface or

the routing instance to which the packet should be sent back when it exits the service engine. The entire packet content is

available, starting from the layer 3 (IP) header. APIs would be used to, for example, rewrite bytes 20 through 40 of a packet, or to

find out if a packet is original data traffic or if it has been duplicated for monitoring. If the application needs to do accurate

time-based calculations for packet events, it may make use of precise hardware timestamping APIs to read a cycle count from the

current CPU, or to read the time at which the hardware received the packet.
8 Copyright ⓒ 2011, Juniper Networks, Inc.
Creating Innovative Embedded Applications in the Network with the Junos SDK
Using Other Junos OS Features on the Service Engine
The Services SDK provides API access to improved memory constructs for custom use by the applications. They are internally used

by other accessible features such as the forwarding database and the policy database.
Two sets of memory management APIs provided in the Services SDK help in this multi-threaded environment designed for low-

latency packet processing. The first consists of a set of shared memory APIs that allocate and deallocate memory from a memory

pool that has a hard-wired TLB entry, thus guaranteeing no TLB misses upon memory accesses. This memory can be shared

among several processes if desired, but it is equally useful within the context of a single process because of the benefit of

consistently fast access times. On top of this shared memory pool, another set of caching APIs provides improves allocation and

deallocation times with a lockless design.
The forwarding database is an optionally configurable feature that makes use of this memory. If it is turned on, the IP routing

and autonomous system information is downloaded to the service engine and installed in this fast-access memory. Because it is

quickly accessible, it is suitable for use in the fast lookups in the inline processing path of the services plane, but it is also

accessible in the control sub-plane.
The policy database is another similar optionally configurable feature. It is also carved out of the fast-access shared memory pool.

The policy database is populated by the applications with their own custom-configured packet-processing rules. It is intended for

use during policy lookups during packet processing, and it is especially useful in construction of service applications, as

described later.
Development with Security and Reliability in Mind
The Junos SDK enables applications to run securely alongside the Junos OS platform’s base applications while taking advantage

of a wide range of its interfaces. First this section looks at security from multiple perspectives. We examine how the operator, the

Junos OS software, and the device are protected and managed, and equally how the application developer can use Junos

features to build additional security into an application. Lastly, we examine how applications can leverage the sophisticated

mechanisms built into the Junos OS for software and hardware reliability and fault tolerance.
Securely Deploying and Operating Applications
All Junos SDK-built software packages are easily installed in the same way as Junos OS packages, and thus installation benefits

from the intrinsic security of the existing process. When the software package originates from a third party, this process demands

explicit configured authorization of the software provider. This authorization is verified in an administrator-controlled list of

third-party providers. Naturally, the active system operator, if not an administrator, must have been granted permission to

perform any software installations.
The Junos operating system further guarantees software authenticity during installation, but also every time any process is

launched. This mechanism safeguards genuineness, tamper resistance, and trustworthiness of the source using well-known

public-key cryptography techniques involving the certification of the provider and a provider-generated signature that is not

forgeable. Certification of third-party providers is securely controlled by Juniper Networks, while tools in the Junos SDK make for

an uncomplicated process for a partner. Furthermore, signatures are generated and incorporated into the software package

along with the certificate automatically during the package build process.
Once an application has been authorized and authenticated successfully, it is executed in a non-root user-space resource-

controlled environment managed by the Junos OS. The kernel provides each process with the familiar protected memory

environment, while ensuring that the process acts within policy-defined resource and access limits. These limits are controlled on

several levels: limits global to the operating system, those that are provider-specific, those configured by the operator, and those

imposed by the application developer. The most restrictive values over all levels are applied. They can control features such as the

familiar UNIX rlimits and access to certain APIs and system calls, and they can give certain permissions normally associated with

the root user.
These mechanisms provide sensible assurances about Junos SDK-built software. Such programs cannot directly interfere or cause

harm to the native Junos OS software or other third party software. An application can, however, make changes with wide-

ranging impacts on the device operation. For example, you must take great care when wielding the power of routing and

filtering, so that resulting changes are in accordance with the expectations of the operator.
Copyright ⓒ 2011, Juniper Networks, Inc. 9
Creating Innovative Embedded Applications in the Network with the Junos SDK
Developing and Managing Additional Security and Resilience
You can build security and resilience into an application in several ways using the tools and APIs of the Junos SDK.
As we have seen, one level of the policy-defined resource limits imposed on Junos SDK-built applications is controlled by the

application developer. In fact, the developer must associate each process with a role having some permissions. Native roles have

predefined permissions, but new roles may also be defined and used to apply customized permissions. In addition to secure

implementation and coding practices, this is one practical way to guarantee by design that an application does not act outside its

intended limits.
We use the Junos health-monitoring framework to deal with failures in applications. This framework allows applications or even

individual threads to be monitored for health by simple echoing of periodic heartbeat messages. Using APIs, an application

developer can optionally register with the health-monitoring framework to control the frequency of these passive heartbeats and

specify an action in the event of a given number of missed echoes. Typically the action is to assume that a failure has occurred

and restart the application. Innate to the Junos OS is the ability to detect and report applications that fail several times (thrash)

upon restart within a short time period. You can also configure other actions such as RE or service engine switchover for critical

applications, enabling a redundant hot-standby environment.
Application High Availability
When a device is provisioned with redundant REs or service engines, the applications can take advantage of their redundancy,

providing varying degrees of application hot-standby support.
On the RE, you can turn on hot standby with graceful routing engine switchover (GRES). To enable high availability in an

application, you can optionally run the application on both the master (active) and backup (standby) REs. The application can

register for notification of RE switchover caused manually or by software or hardware faults. APIs built on top of the Junos SDK’s

IPC libraries allow state replication between the master and the backup application, so that the standby instance, upon

switchover, can pick up where the previously active instance had left off.
Simpler high-availability support is also available for RE-based applications when GRES is enabled. In this design the application

need not run on both REs simultaneously. It can make use of APIs to store opaque binary data in semi-persistent storage. While

GRES is enabled, these data stores are automatically replicated to the backup RE. Using this functionality, upon a switchover the

application is launched on the new master RE, and it can easily pull its data back out of storage to pick up where it formerly left

off. Furthermore, in the context of a single RE, if an application is restarted for any reason, it can equally resume its previous

state if desired.
Finally, the service engines themselves also support a mirrored redundant configuration. Rapid switchover behavior is all the more

critical to mitigate packet loss, so applications installed on service engines can be augmented to support hot standby. As when

running an application on two REs simultaneously, applications can use the same APIs to replicate state from the master to the

backup.
Building a Service Application
In this section we describe the design considerations in building a service application (one that processes transit data traffic in

the services plane) with the Junos SDK. We examine methods of steering the data traffic to a service engine, and we look at the

necessary architectural components of a complete solution. Finally, we look at the different approaches in the services plane for

receiving data traffic and building a service application.
Managing the Services Plane and Data Traffic Steering
As we have described above, the control plane plays a large role in managing the software of the data and services planes. To

this end, the control plane must work in three key ways to create a functioning service application.
First, service application software must be installed on targeted service engines. Based on configuration, a Junos OS daemon on

the RE pushes the software onto each service engine before it can be run, since the service modules have no disk. This step

could involve installing different software on different service engines, or the same software if scaling or redundancy is required.

On security- and service-oriented devices, all the service engines run the same software, eliminating configuration of applications

per service engine.
10 Copyright ⓒ 2011, Juniper Networks, Inc.
Creating Innovative Embedded Applications in the Network with the Junos SDK
Second, once the service software starts on a service engine, to the software must be configured with policies to apply, and it

may want eventually to report back information through the Junos UI. Policies in this context are simply the configuration

pertaining to the service application, especially its packet processing. The model used to achieve the coordination of policy and

reporting is to have a central management component (daemon) on the RE’s control plane. This daemon coordinates multiple

software instances in the case of multiple service engines, and it handles RE-specific functionality such as loading custom

configuration and processing changes. It also communicates the relevant policy information to the corresponding service

applications connecting to it. Communication can flow in the other direction as well if a service application, for example, wanted

to send statistics for eventual display in response to an operational command.
Third, the management component optionally controls the data plane to steer packets to the services plane for servicing on

service engines. We examine three of several approaches to doing so. These approaches are actually only pertinent to

forwarding-oriented devices, where the service plane is optional. In security- and service-oriented devices, the services plane is

always in the packet processing path, so no steering is implicit.
Routes are the simplest steering approach. A route to a service engine is a route like any other, containing a prefix against which

to match and a next hop; but the next hop is an interface managing a service engine, as opposed to an external address. Once

this route is installed in the forwarding table, packets matching it are redirected to the service engine. Variants on basic routing

are also possible. For example, equal-cost multipath routing and filter-based forwarding are both interesting options that can

include routes to service engines. Routes can also be controlled programmatically with the Junos SDK’s APIs.
The second option is to create and apply service sets to network interfaces on ingress or egress. A service set is a

conglomeration of an identifier, one or more services’ policies to be applied, and a service engine to which to redirect packets for

servicing. When a service set is applied to an interface, the data plane selects packets flowing through that interface as part of

the service set, and accordingly steers them to the specified service engine for servicing. The service application can retrieve and

correlate a packet’s service set identifier in order to apply the correct policy configured in the service set. When traffic is selected

for servicing at a network interface, the service set is said to be an interface-style service set. Other service set styles are equally

useful for steering data traffic to the service engine, but this style in particular allows the traffic to be filtered so that only a

subset of the traffic is steered towards the service engine for servicing. This filtering is done with service filters based on Junos

stateless firewall filters.
A third option uses the Junos OS sampling framework. This feature of the PFE duplicates packets according to a configured

frequency and sample length. All packets that enter the PFE’s sampling engine do so as a result of the application of a stateless

firewall filter. These duplicate packets typically are steered to a service engine for service processing. Given that the original

packet is forwarded without impact and is not modifiable by the service, sampling is well suited to monitoring-style services.

Service sets can also be combined with sampling when a service chain is needed, as described in the next section.
These options, in combination with other Junos features, make for a large array of possibilities in the design of a service solution.
The Design Models and Components
In the previous section we described the management component that coordinates the service applications running on the

service engine. This section introduces two more components of a service application and two models available for building them

with the Services SDK. An individual service engine supports service applications of only one model at a time. However, the

service engine supports any variation of non-packet-processing applications running in its control plane.
Autonomy and Polling Control Enabled by the Process Model
In this model, the service engine supports a sole service application consisting of a single two-component daemon, as in the

figure below. The control component communicates with the management component, usually via the IPC APIs. At a minimum, it

receives and stores the configured service policies, but sending statistics and status information is commonplace as well. The

control component is so named because it deals exclusively with control traffic, whether it be IPC or any communication outside

the device.
Copyright ⓒ 2011, Juniper Networks, Inc. 11
Creating Innovative Embedded Applications in the Network with the Junos SDK
The data component uses these stored policies and performs the servicing on the data traffic, hence its name. This component

spans many real-time software threads, each tied to a single and exclusively used hardware thread of the service engine’s CPU.

We call such a real-time thread executing a packet-polling loop a data thread. During the startup phase before these threads are

started, the Services SDK and kernel set up a series of zero-copy input and output first-in-first-out (FIFO) queue pairs, where

eventually one data thread is tied to and services one pair of queues in its loop. Generally the steps of each loop are to receive a

packet from the input queue if one is available, process it, and enqueue it in its output queue. Processing is based on either the

service policies stored in the memory shared with the control component or the application’s per-flow or per-session state.
Figure 2. Services SDK daemon and traffic flow
Part of the Services SDK provides the APIs that assist the setup of this multithreaded environment to work with the hardware

threads of the CPU. As described above, it also includes fast-access shared-memory APIs, locking constructs suitable to the

environment, and packet manipulation functions.
Service Chaining and Session Management Enabled by the Plug-in Model
The Services SDK provides the plug-in model to allow several data-traffic-oriented services to coexist and even cooperate through

an event framework within a single service engine. Plug-ins can send and receive both synchronous and asynchronous custom

events in a loosely coupled way, but this section focuses on the standard plug-in events. Furthermore, this model facilitates

maintaining the service policies in the policy database and accessing flow- or session-based state. Also, IP fragments are

reassembled automatically before processing.
Figure 3. Services SDK plug-in and data traffic flow
REPFEControlcomponentService EnginePacket FIFO Queues:Unserviced Local or Internal Control TracTransit Data TracUnserviced

Transit TracDatacomponentManagementcomponentData Threads• • •ControlcomponentService EngineDatacomponentTransit Data

Trac• • •• • •CDCDCD
12 Copyright ⓒ 2011, Juniper Networks, Inc.
Creating Innovative Embedded Applications in the Network with the Junos SDK
In this model, a Junos OS platform daemon managing the plug-ins starts and creates its own data component to run one or

multiple plug-ins linked into it as in the figure above. The plug-ins are implemented as shared libraries. Each has an entry

function that, when called, registers callback functions to serve as the control and data event handlers. Control events, called

from non-packet-processing real-time threads, initialize the plug-in and update policies originated from a management

component. Real-time data threads are managed by the Junos daemon managing the plug-ins. Each data thread internally polls

for packets and dispatches them as data events serially through a chain of plug-ins. This model works with the service set

steering method, which allows for a service order to be specified by the operator per service set, and this order ultimately

determines the order of packet delivery, and hence service processing.
With every packet of a session (based on the classic 5-tuple) dispatched as a data event, a per-session context to store custom

session state is delivered as supplementary metadata. This state expedites processing if plug-ins store policy decisions in the state

once they have been looked up for the first packet of a session.
This model has numerous advantages for some styles of service applications. First, if a service is flow or session based, the

framework sets up the packet-polling loops (data threads), provides a session context with every packet, and provides session

open and close events. Services from multiple providers (Juniper Networks and third parties) can run on a single service engine

to collaborate in a configurable order.
On the security- and service-oriented devices, the service engines always run in this plug-in model. On the forwarding-oriented

devices, while the service engines support both models, this plug-in model supports pairing two service engines together, where

one is standing by with replicated state information in the event of a failure. Overall, this model generally provides more features

and facilitates rapid portable development where session-based packet processing is the main focus.
Virtual Build Environment
The Junos SDK provides a Virtual Build Environment (VBE) that creates a comprehensive and consistent way to develop

applications in just about any operating system environment you choose. The VBE is, in actuality, a FreeBSD operating system

image that is executable inside VMware Player, which is freely available software that runs on a wide range of operating systems.

This approach allows for a clean separation of a portable and duplicable build environment. Samba comes pre-configured in the

VBE, so that sharing the file system with that of the native system is easy to achieve. Thus, developers are free to use any

development environment software they choose.
The APIs and build tools of the Junos SDK are built and tested at the same time as the rest of the Junos OS, and they will

continue to be enhanced in step with the Junos OS release schedule. Developers manage the APIs and tools in the VBE by

installing packages of these APIs and tools matching the Junos OS versions for which they wish to work.
The Toolchain package contains development tools such as the compilers, linkers, debuggers, bmake, and other tools that are

used by the scripts that invoke the application build process. The Backing Sandbox package contains the actual libraries, APIs,

and scripts with which to build the applications. It also contains a growing number of sample applications that demonstrate the

API usage.
Various scripts from the backing sandbox greatly assist with the steps of development. For example, in the first required step, a

script is used to create a private signing key and certificate request. Once the partner is issued a certificate, the partner developer

can begin creating development sandboxes. This creation is again automated with a script. These sandboxes are directories in

which the developer creates applications. Sandboxes have a skeleton of useful subdirectories and build scripts, but they can

optionally be created to contain one or more sample applications as well.
Inside a development sandbox are two key top-level scripts. One of these invokes a build to compile the code into binaries, and

the other invokes packaging. The packaging can build multiple packages where the developer defines what goes into each one.

All of the generated packages automatically include the certificate and a signature. Packages can also contain other packages to

create easily deployable bundles. They are placed in one subdirectory of the sandbox and can be copied to the device for

installation. You can debug applications in their sandbox and on the device. In the VBE one can use the well-known GNU

debugger (gdb) to do post-mortem debugging with a core file or to attach to a live instance of gdbserver on the device.
Copyright ⓒ 2011, Juniper Networks, Inc. 13
Creating Innovative Embedded Applications in the Network with the Junos SDK
Application Examples
In this section we describe two real sample applications designed and implemented with the RE and Services SDKs. We start with

brief descriptions of the applications at a high level, focusing on important practical topics such as the useful software libraries

and data traffic steering. In turn we explain the creation of MoniTube, an IPTV monitoring and mirroring application; and

Equilibrium, a load-balancing and traffic classification application.
MoniTube
MoniTube is an application that can be used to monitor the quality of IPTV streams at any point where it is deployed on the

network. It can report several basic quality-indicating metrics. MoniTube can also mirror IPTV streams to other locations.
When MoniTube is installed, the Junos OS user interface is seamlessly extended to allow for its configuration and for reporting of

the video streams’ quality through operational commands. Configurations can then be entered to separately identify the streams

of interest for monitoring and mirroring. A MoniTube management application runs on the RE to load this configuration and

transfer it to a MoniTube service application running on a service engine.
The MoniTube application needs to see all packets of the streams it is to monitor and mirror, with no need to alter the original

streams. Therefore, the application’s packet processing functionality runs on a service engine that does not receive the original

streams, but rather copies of all its packets for monitoring.
To receive copies of packets, we take advantage of the Junos OS sampling feature, which enables an operator to send sampled

packets to a service engine, where service applications can process them. Our application needs to receive all packets selected for

sampling, so the sampling rate is configured as 1, indicating that every packet steered to the sampling framework is sampled to

the target service engine, and hence to the MoniTube service.
We use programmatically controlled firewall filters to select packets for which sampling is required. These filters narrow the traffic

selected for servicing. For example, if a variety of traffic flowed into the device through a given interface, a filter could be set up

to match all UDP traffic in 226.16.64.0/18 and 228.1.2.3/32 and select it to be sampled, and thus directed to MoniTube. With

programmatic control over the filters, the selection of traffic for sampling could be more dynamic if needed. Our simple approach

gives the operator the flexibility to select all or a subset of the device’s interfaces and traffic to be under MoniTube’s

management.
The MoniTube service is implemented as a multithreaded daemon running on the service engine. Thus, it follows the Services

SDK’s process model, but a plug-in model implementation of the MoniTube application is present in the Junos SDK sample

applications for comparison. In the process-model edition, the daemon’s main thread (control component) is primarily responsible

for communication with the management component; but more interesting are its many other real-time data threads dedicated to

packet processing. These threads poll the input packet queues, dequeue packets, perform the monitoring calculations, do any

packet manipulation for mirroring (e.g. changing the destination address), and then if the packet is not mirrored, dispose of the

packet, since it is a duplicate of the real selected traffic. If the packet is to be mirrored, we send it as modified during the

processing, enqueuing it into an output queue. This application architecture can easily be applied to dealing with real packets

instead of copies, where the original monitored packet is sent by default and not dropped. However, with sampled traffic, we

know we will never adversely impact the original IPTV streams.
Equilibrium
Equilibrium is an application that provides two simple functionalities that are implemented and potentially deployed as separate

services: load balancing and traffic classification.
The load balancing service has some configured facade addresses, and for each has a pool of real addresses. When it sees traffic

destined to a facade address, the service redirects it to an address from the pool with the least load, based on the number of

connections. The classification service provides the ability to redirect traffic to a single different destination address, but based on

a match with a destination port number rather than a destination (facade) address.
Once installed, just as with MoniTube, a management component runs on the RE, reading in any configuration for each of these

services. It passes this information down to the service’s control component running on the service engine. Multiple service

engines may also be configured to run one or both of the Equilibrium services, which work through service sets. The

management component is also responsible for gathering this information and for sending only the required policies to the

Equilibrium service running on each module.
14 Copyright ⓒ 2011, Juniper Networks, Inc.
Creating Innovative Embedded Applications in the Network with the Junos SDK
2000378-001-EN Feb 2011
Copyright 2011 Juniper Networks, Inc. All rights reserved. Juniper Networks, the Juniper Networks logo, Junos,
NetScreen, and ScreenOS are registered trademarks of Juniper Networks, Inc. in the United States and other
countries. All other trademarks, service marks, registered marks, or registered service marks are the property of
their respective owners. Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper
Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice.
EMEA Headquarters
Juniper Networks Ireland
Airside Business Park
Swords, County Dublin, Ireland
Phone: 35.31.8903.600
EMEA Sales: 00800.4586.4737
Fax: 35.31.8903.601
APAC Headquarters
Juniper Networks (Hong Kong)
26/F, Cityplaza One
1111 King’s Road
Taikoo Shing, Hong Kong
Phone: 852.2332.3636
Fax: 852.2574.7803
Corporate and Sales Headquarters
Juniper Networks, Inc.
1194 North Mathilda Avenue
Sunnyvale, CA 94089 USA
Phone: 888.JUNIPER (888.586.4737)
or 408.745.2000
Fax: 408.745.2100
www.juniper.net
Printed on recycled paper
To purchase Juniper Networks solutions,
please contact your Juniper Networks
representative at 1-866-298-6428 or
authorized reseller.
The motivation for using the Junos OS service sets was twofold. First, unlike the case of MoniTube, both services need to act
on real traffic transiting the device. A service set specifies an ordered set of services with this requirement, and one or more
sets may be associated with a service engine. Data traffic is redirected to the service set when filtering on interfaces detects
matched packets that need to be serviced. Second, we want to allow Equilibrium services to be run together on one service
engine, and potentially with other Junos or third-party services. The plug-in model achieves this aim: in general, plug-ins are
meant to work with service sets.
In accordance with the configured order in the service set, packets pass through the data event handlers of the Equilibrium
services and any other services configured in the same set. When seeing the first packet of a session, we look into our
configuration policies for facade address or port matches and store the corresponding action in the session context. For
subsequent packets of the same session, this action is available in the context and is immediately applied. This method
expedites servicing, as we simply retrieve the original action taken on packets of each flow.
Both MoniTube and Equilibrium are sample applications provided with the Junos SDK, where all code, build, and deployment
instructions are provided.
Conclusion
The Junos SDK enables applications to be embedded in the network by running them in the devices themselves on top of the
Junos OS. Juniper Networks uses the Junos SDK extensively to build its own software components and service applications,
and many customers, partners, and research organizations are already doing the same.
The Junos SDK enables network element programmability, but Juniper Networks views the Junos OS as a platform, not
just for such programmability, but as a host to innovative applications that allow differentiation to create value and deliver
tomorrow’s solutions.
About Juniper Networks
Juniper Networks is in the business of network innovation. From devices to data centers, from consumers to cloud providers,
Juniper Networks delivers the software, silicon and systems that transform the experience and economics of networking.
The company serves customers and partners worldwide. Additional information can be found at www.juniper.net.
Disclaimer
This work is an excerpt from the forthcoming book “Network Embedded Management and Applications” expected to be
published in 2011 by Springer Press.