.. This document is formatted using reStructuredText, which is a Markup

   Syntax and Parser Component of Docutils for Python.  An html version

   of this document can be generated using the following command:

     rst2html.py doc/parallel-linked-images.txt >doc/parallel-linked-images.html

======================

Parallel Linked Images

======================

:Author: Edward Pilatowicz

:Version: 0.1

Problems

========

Currently linked image recursion is done serially and in stages.  For

example, when we perform an "pkg update" on an image then for each child

image we will execute multiple pkg.1 cli operations.  The multiple pkg.1

invocations on a single child image correspond with the following

sequential stages of pkg.1 execution:

1) publisher check: sanity check child publisher configuration against

   parent publisher configuration.

2) planning: plan fmri and action changes.

3) preparation: download content needed to execute planned changes.

4) execution: execute planned changes.

So to update an image with children, we invoke pkg.1 four times for each

child image.  This architecture is inefficient for multiple reasons:

- we don't do any operations on child images in parallel

- when executing multiple pkg.1 invocations to perform a single

  operation on a child image, we are constantly throwing out and

  re-initializing lots of pkg.1 state.

To make matters worse, when as we execute stages 3 and 4 on a child

image the pkg client also re-executes previous stages.  For example,

when we start stage 4 (execution) we re-execute stages 2 and 3.  So for

each child we update we end up invoking stage 2 three times, and stage 3

twice.  This leads to bugs like 18393 (where it seems that we download

packages twice).  It also means that we have caching code buried within

the packaging system that attempts to cache internal state to disk in an

effort to speed up subsequent re-runs of previous stages.

Solutions

=========

Eliminate duplicate work

------------------------

We want to eliminate a lot of the duplicate work done when executing

packaging operations on children in stages.  To do this we will update

the pkg client api to allow callers to:

- Save an image plan to disk.

- Load an image plan from disk.

- Execute a loaded plan from disk without first "preparing" it.  (This

  assumes that the caller has already "prepared" the plan in a previous

  invocation.)

In addition to eliminating duplicated work during staged execution, this

will also allow us to stop caching intermediate state internally within

the package system.  Instead client.py will be enhanced to cache the

image plan and it will be the only component that knows about "staging".

To allow us to save and restore plans, all image plan data will be saved

within a PlanDescription object, and we will support serializing this

object into a json format.  The json format for saved image plans is an

internal, unstable, and unversioned private interface.  We will not

support saving an image plan to disk and then executing it later with a

different version of the packaging system on a different host.  Also,

even though we will be adding data into the PlanDescription object we

will also not be exposing any new information about an image plan to via

the PlanDescription object to api consumers.

An added advantage of allowing api consumers to save an image plan to

disk is that it should help with our plans to have the api.gen_plan_*()

functions to be able to return PlanDescription object for child images.

A file descriptor (or path) associated with a saved image plan would be

one way for child images to pass image plans back to their parent (which

could then load them and yield them as results to api.gen_plan_*()).

Update children in parallel

---------------------------

We want to enhance the package client so that it can update child images

in parallel.

Due to potential resource constraints (cpu, memory, and disk io) we

cannot entirely remove the ability to operate on child images serially.

Instead, we plan to allow for a concurrency setting that specifies how

many child images we are willing to update in parallel.  By default when

operating on child images we will use a concurrency setting of 1, this

maintains the current behavior of the packaging system.  If a user wants

to specify a higher concurrency setting, they can use the "-C N" option

to subcommands that recurse (like "install", "update", etc) or they can

set the environment variable "PKG_CONCURRENCY=N".  (In both cases N is

an integer which specifies the desired concurrency level.)

Currently, pkg.1 worker subprocesses are invoked via the pkg.1 cli

interfaces.  When switching to parallel execution this will be changed

to use a json encoded rpc execution model.  This richer interface is

needed to allow worker processes to pause and resume execution between

stages so that we can do multi-staged operations in a single process.

Unfortunately, the current implementation does not yet retain child

processes across different stages of execution.  Instead, whenever we

start a new stage of execution, we spawn one process for each child

images, then we make a remote procedure call into N images at once

(where N is our concurrency level).  When an RPC returns, that child

process exits and we start a call for the next available child.

Ultimately, we'd like to move to model where we have a pool of N worker

processes, and those processes can operate on different images as

necessary.  These processes would be persistent across all stages of

execution, and ideally, when moving from one stage to another these

processes could cache in memory the state for at least N child images so

that the processes could simply resume execution where they last left

off.

The client side of this rpc interface will live in a new module called

PkgRemote.  The linked image subsystem will use the PkgRemote module to

initiate operations on child images.  One PkgRemote instance will be

allocated for each child that we are operating on.  Currently, this

PkgRemote module will only support the sync and update operations used

within linked images, but in the future it could easily be expanded to

support other remote pkg.1 operations so that we can support recursive

linked image operations (see 7140357).  When PkgRemote invokes an

operation on a child image it will fork off a new pkg.1 worker process

as follows:

	pkg -R /path/to/linked/image remote --ctlfd=5

this new pkg.1 worker process will function as an rpc server which the

client will make requests to.

Rpc communication between the client and server will be done via json

encoded rpc.  These requests will be sent between the client and server

via a pipe.  The communication pipe is created by the client, and its

file descriptor is passed to the server via fork/exec.  The server is

told about the pipe file descriptor via the --ctlfd parameter.  To avoid

issues with blocking IO, all communication via this pipe will be done by

passing file descriptors.  For example, if the client wants to send a

rpc request to the server, it will write that rpc request into a

temporary file and then send the fd associated with the temporary file

over the pipe.  Any reply from the server will be similarly serialized

and then sent via a file descriptor over the pipe.  This should ensure

that no matter the size of the request or the response, we will not

block when sending or receiving requests via the pipe.  (Currently, the

limit of fds that can be queued in a pipe is around 700.  Given that our

rpc model includes matched requests and responses, it seems unlikely

that we'd ever hit this limit.)

In the pkg.1 worker server process, we will have a simple json rpc

server that lives within client.py.  This server will listen for

requests from the client and invoke client.py subcommand interfaces

(like update()).  The client.py subcommand interfaces were chosen to be

the target for remote interfaces for rpc calls for the following

reasons:

- Least amount of encoding / decoding.  Since these interfaces are

  invoked just after parsing user arguments, they mostly involve simple

  arguments (strings, integers, etc) which have a direct json encoding.

  Additionally, the return values from these calls are simple return

  code integers, not objects, which means the results are also easy to

  encode.  This means that we don't need lots of extra serialization /

  de-serialization logic (for things like api exceptions, etc).

- Output and exception handling.  The client.py interfaces already

  handle exceptions and output for the client.  This means that we don't

  have to create new output classes and build our own output and

  exception management handling code, instead we leverage the existing

  code.

- Future recursion support.  Currently when recursing into child images

  we only execute "sync" and "update" operations.  Eventually we want to

  support pkg.1 subcommand recursion into linked images (see 7140357)

  for many more operations.  If we do this, the client.py interfaces

  provide a nice boundary since there will be an almost 1:1 mapping

  between parent and child subcommand operations.

Child process output and progress management

--------------------------------------------

Currently, since child execution happens serially, all child images have

direct access to standard out and display their progress directly there.

Once we start updating child images in parallel this will no longer be

possible.  Instead, all output from children will be logged to temporary

files and displayed by the parent when a child completes a given stage

of execution.

Additionally, since child images will no longer have access to standard

out, we will need a new mechanism to indicate progress while operating

on child images.  To do this we will have a progress pipe between each

parent and child image.  The child image will write one byte to this

pipe whenever one of the ProgressTracker`*_progress() interfaces are

invoked.  The parent process can read from this pipe to detect progress

within children and update its user visible progress tracker

accordingly.