\documentclass[10pt,a4paper,english]{article}
\usepackage{babel}
\usepackage{ae}
\usepackage{aeguill}
\usepackage{shortvrb}
\usepackage[latin1]{inputenc}
\usepackage{tabularx}
\usepackage{longtable}
\setlength{\extrarowheight}{2pt}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{color}
\usepackage{multirow}
\usepackage{ifthen}
\usepackage[colorlinks=true,linkcolor=blue,urlcolor=blue]{hyperref}
\usepackage[DIV12]{typearea}
%% generator Docutils: http://docutils.sourceforge.net/
\newlength{\admonitionwidth}
\setlength{\admonitionwidth}{0.9\textwidth}
\newlength{\docinfowidth}
\setlength{\docinfowidth}{0.9\textwidth}
\newlength{\locallinewidth}
\newcommand{\optionlistlabel}[1]{\bf #1 \hfill}
\newenvironment{optionlist}[1]
{\begin{list}{}
  {\setlength{\labelwidth}{#1}
   \setlength{\rightmargin}{1cm}
   \setlength{\leftmargin}{\rightmargin}
   \addtolength{\leftmargin}{\labelwidth}
   \addtolength{\leftmargin}{\labelsep}
   \renewcommand{\makelabel}{\optionlistlabel}}
}{\end{list}}
\newlength{\lineblockindentation}
\setlength{\lineblockindentation}{2.5em}
\newenvironment{lineblock}[1]
{\begin{list}{}
  {\setlength{\partopsep}{\parskip}
   \addtolength{\partopsep}{\baselineskip}
   \topsep0pt\itemsep0.15\baselineskip\parsep0pt
   \leftmargin#1}
 \raggedright}
{\end{list}}
% begin: floats for footnotes tweaking.
\setlength{\floatsep}{0.5em}
\setlength{\textfloatsep}{\fill}
\addtolength{\textfloatsep}{3em}
\renewcommand{\textfraction}{0.5}
\renewcommand{\topfraction}{0.5}
\renewcommand{\bottomfraction}{0.5}
\setcounter{totalnumber}{50}
\setcounter{topnumber}{50}
\setcounter{bottomnumber}{50}
% end floats for footnotes
% some commands, that could be overwritten in the style file.
\newcommand{\rubric}[1]{\subsection*{~\hfill {\it #1} \hfill ~}}
\newcommand{\titlereference}[1]{\textsl{#1}}
% end of "some commands"
\title{Cpuset Library and Linux Kernel Support}
\author{}
\date{}
\hypersetup{
pdftitle={Cpuset Library and Linux Kernel Support}
}
\raggedbottom
\begin{document}
\maketitle


\setlength{\locallinewidth}{\linewidth}
% rst3: filename: libcpuset.tex 
\pagenumbering{roman}\large\parindent0pt\parskip10pt
This document describes the 'C' library \textbf{libcpuset} interface to Linux cpusets.

Cpusets provide system-wide control of the CPUs on which tasks may execute, and
the memory nodes on which they allocate memory. Each cpuset defines a list of
allowed CPUs and memory nodes, and each process in the system is attached to a
cpuset. Cpusets are represented in a hierarchical virtual file system. Cpusets
can be nested and they have file-like permissions.

The efficient administration of large multi-processor systems depends on
dynamically allocating portions of the systems CPU and memory resources to
different users and purposes. The optimum performance of NUMA systems depends
optimizing CPU and memory placement of critical applications, and minimizing
interference between applications. Cpusets provides a convenient means to
control such CPU and memory placement and usage.
\begin{quote}
\begin{description}
\item [Author:]
Paul Jackson


\item [Address:]
\href{mailto:pj@sgi.com}{pj@sgi.com}


\item [Date:]
14 November 2006


\item [Copyright:]
Copyright (c) 2006-2007 SGI.  All Rights Reserved.


\end{description}
\end{quote}
\newpage\small \vspace*{1cm}
Permission is hereby granted, free of charge, to any person obtaining a copy of
this Documentation, to deal in the Documentation without restriction, including
without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Documentation, and to permit persons to
whom the Documentation is furnished to do so, subject to the following
conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Documentation.

THE DOCUMENTATION IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL SILICON
GRAPHICS, INC. BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE DOCUMENTATION OR THE USE OR OTHER DEALINGS IN THE DOCUMENTATION.

Except as contained in this notice, the names of Silicon Graphics and SGI shall
not be used in advertising or otherwise to promote the sale, use or other
dealings in this Documentation without prior written authorization from SGI.


%___________________________________________________________________________
\hspace*{\fill}\hrulefill\hspace*{\fill}


This document is written using the outline processor \href{http://webpages.charter.net/edreamleo/front.html}{Leo}, and version controlled
using \href{http://cssc.sourceforge.net/index.shtml}{CSSC}. It is rendered using \href{http://docutils.sourceforge.net}{Python Docutils} on \href{http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html}{reStructuredText}
extracted from \href{http://webpages.charter.net/edreamleo/front.html}{Leo}, directly into both \href{http://www.w3.org/MarkUp}{html} and \href{http://www.latex-project.org}{LaTeX}.  The \href{http://www.latex-project.org}{LaTeX} is
converted into \href{http://www.adobe.com/products/acrobat}{pdf} using the \href{http://www.tug.org/applications/pdftex}{pdflatex} utility.  The \href{http://www.w3.org/MarkUp}{html} is converted into
plain text using the \href{http://lynx.isc.org}{lynx} utility.


%___________________________________________________________________________
\hspace*{\fill}\hrulefill\hspace*{\fill}


Silicon Graphics and SGI are registered trademarks of Silicon Graphics, Inc., in
the United States and other countries worldwide. Linux is a registered
trademark of Linus Torvalds in several countries. Novell is a registered
trademark, and SUSE is a trademark of Novell, Inc. in the United States and
other countries. All other trademarks mentioned herein are the property of their
respective owners.
\newpage\large\vspace*{1cm}\hypertarget{table-of-contents}{}
\pdfbookmark[0]{Table of Contents}{table-of-contents}
\subsubsection*{~\hfill Table of Contents\hfill ~}
\begin{list}{}{}
\item {} \href{\#why-cpusets}{1~~~Why Cpusets?}

\item {} \href{\#linux-cpuset-kernel-support}{2~~~Linux Cpuset Kernel Support}

\item {} \href{\#using-cpusets-at-the-shell-prompt}{3~~~Using Cpusets at the Shell Prompt}

\item {} \href{\#cpuset-programming-model}{4~~~Cpuset Programming Model}

\item {} \href{\#cpus-and-memory-nodes}{5~~~CPUs and Memory Nodes}

\item {} \href{\#extensible-api}{6~~~Extensible API}

\item {} \href{\#cpuset-text-format}{7~~~Cpuset Text Format}

\item {} \href{\#basic-cpuset-library-functions}{8~~~Basic Cpuset Library Functions}

\item {} \href{\#using-cpusets-with-hyper-threads}{9~~~Using Cpusets with Hyper-Threads}

\item {} \href{\#advanced-cpuset-library-functions}{10~~~Advanced Cpuset Library Functions}

\item {} \href{\#system-error-numbers}{11~~~System Error Numbers}

\item {} \href{\#change-history}{12~~~Change History}

\end{list}

\newpage \pagenumbering{arabic} \normalsize

%___________________________________________________________________________

\hypertarget{why-cpusets}{}
\pdfbookmark[0]{1~~~Why Cpusets?}{why-cpusets}
\section*{1~~~Why Cpusets?}

The essential purpose of cpusets is to provide CPU and memory containers or
``soft partitions'' within which to run sets of related tasks.

On an SMP (multiple CPU) system without some means of CPU placement, any task
can run on any CPU. On a NUMA (multiple memory node) system, any memory page can
be allocated on any node. This can cause both poor cache locality and poor
memory access times, substantially reducing performance and run-time
repeatability. By restraining all other jobs from using any of the CPUs or
memory nodes assigned to critical jobs, interference with critical jobs can be
minimized.

For example, some multi-threaded high performance computing (HPC) jobs consist
of a number of threads that communicate via message passing interfaces (MPI) and
other such jobs rely on multi-platform shared-memory parallel programming
(OpenMP) that can tightly couple parallel computation threads using special
language directives. It is common that threads in such jobs need to be executing
at the same time to make optimum progress. In such cases, if a single thread
loses a CPU, all threads stop making forward progress and spin at a barrier.
Cpusets can eliminate the need for a gang scheduler, provide isolation of
one such job from other tasks on a system, and facilitate providing equal
resources to each thread in such a job.  This results in both optimum and
repeatable performance.

This document focuses on the 'C' API provided by the user level \textbf{libcpuset}
library. This library depends on the following Linux 2.6 kernel facilities:
\begin{quote}
\begin{itemize}
\item {} 
sched{\_}setaffinity (for CPU binding),

\item {} 
mbind and set{\_}mempolicy (for memory node binding), and

\item {} 
kernel cpusets support.

\end{itemize}
\end{quote}

The sched{\_}setaffinity, mbind and set{\_}mempolicy calls enable specifying
the CPU and memory placement for individual tasks.  On smaller or limited
use systems, these calls may be sufficient.

The kernel cpuset facility provides additional support for system wide
management of CPU and memory resources, by related sets of tasks.
It provides a hierarchical structure to the resources, with file system
like name-space and permissions, and support for guaranteed exclusive use
of resources.

The Linux kernel provides the following support for cpusets:
\begin{quote}
\begin{itemize}
\item {} 
Each task has a link to a cpuset structure that specifies the
CPUs and memory nodes available for its use.

\item {} 
A hook in the sched{\_}setaffinity and mbind system calls ensures that
any requested CPU or memory node is available in that tasks cpuset.

\item {} 
Tasks sharing the same placement constraints reference the same cpuset.

\item {} 
These kernel cpusets are arranged in a hierarchical virtual file system,
reflecting the possible nesting of ``soft partitions''.

\item {} 
The kernel task scheduler is constrained to only schedule a task on
the CPUs in that task's cpuset.

\item {} 
The kernel memory allocator is constrained to only allocate physical
memory to a task from the memory nodes in that tasks cpuset.

\item {} 
The kernel memory allocator provides an economical per-cpuset metric
of the aggregate memory pressure (frequency of requests for a free
memory page not easily satisfied by an available free page) of the tasks
in a cpuset (see the per-cpuset 'memory{\_}pressure' file.)

\item {} 
The kernel memory allocator provides the option to request that memory
pages used for file I/O (the kernel page cache) and associated kernel
data structures for file inodes and directories be evenly spread across
all the memory nodes in a cpuset, rather than preferentially allocated on
whatever memory node the task that first accessed the page was first running
(see the per-cpuset 'memory{\_}spread{\_}page' and 'memory{\_}spread{\_}slab' files).

\item {} 
The memory migration facility in the kernel can be controlled using
per-cpuset files, so that when the memory nodes allowed to a task by
cpusets changes, any pages it had on no longer allowed nodes are migrated
to nodes now allowed.

\end{itemize}
\end{quote}

A cpuset constrains the jobs (set of related tasks) running in it to a subset of
the systems memory and CPUs. They enable administrators and system service
software to:
\begin{quote}
\begin{itemize}
\item {} 
Create and delete named cpusets.

\item {} 
Decide which CPUs and memory nodes are available to a cpuset.

\item {} 
Attach a task to a particular cpuset.

\item {} 
Identify all tasks sharing the same cpuset.

\item {} 
Exclude any other cpuset from overlapping a given cpuset,
giving the tasks running in that cpuset exclusive use
of those CPUs and memory nodes.

\item {} 
Perform bulk operations on all tasks associated with a cpuset,
such as varying the resources available to that cpuset, or
hibernating those tasks in temporary favor of some other job.

\item {} 
Perform sub-partitioning with hierarchical permissions and
resource management.

\end{itemize}
\end{quote}

Cpusets are exposed by the kernel to user space by mounting the \texttt{cpuset}
virtual file system (VFS) at \texttt{/dev/cpuset}, rather than by additional
system calls. Such a VFS is a natural way to represent nested resource
allocations and the associated hierarchical permission model.

Within a single cpuset, other facilities such as \texttt{dplace},
first-touch memory placement, pthreads, \texttt{sched{\_}setaffinity} and \texttt{mbind}
can be used to manage processor and memory placement to a more fine-grained level.

There is a single set of kernel mechanisms that supports all these
facilities and provides a consistent processor and memory model
regardless of what mix of utilities and API's you use to manage it.
This provides a consistent execution model for all users.


%___________________________________________________________________________

\hypertarget{linux-cpuset-kernel-support}{}
\pdfbookmark[0]{2~~~Linux Cpuset Kernel Support}{linux-cpuset-kernel-support}
\section*{2~~~Linux Cpuset Kernel Support}

Several developers in the Open Source community provided the Linux 2.6
kernel support for CPU and memory placement, including the following:
\begin{quote}
\begin{itemize}
\item {} 
Robert Love - \texttt{Tech9, Novell} (USA) - Scheduler attributes such as CPU affinity

\item {} 
Simon Derr - \texttt{Bull} (France) - CPU placement, core architecture
and file system interface to cpusets

\item {} 
Andi Kleen - \texttt{SUSE} (Germany) - NUMA memory placement, mbind and mempolicy

\item {} 
Paul Jackson - \texttt{SGI} (USA) - kernel bitmask improvements, kernel cpuset
integration, \textbf{libbitmask}, \textbf{libcpuset}

\end{itemize}
\end{quote}

At least a couple of command line utilities have been developed that use these
affinity calls to allow placing a process on a specific CPU. Robert Love has a
package \emph{schedutils} with a command \texttt{taskset}. The \texttt{numactl} command in Andi
Kleen's work has options to run a specified command on specified CPUs and memory
nodes.

Andi Kleen led a session at the 2003 Kernel Summit in Ottawa in NUMA
memory management, and has been developing a NUMA library and kernel
support for memory placement. Minutes from that session are available
at: \href{http://lwn.net/Articles/40626/}{http://lwn.net/Articles/40626/}.

The cpuset kernel facility and file system for Linux 2.6 kernels is based on the
work of Simon Derr, with integration and refinements by Paul Jackson. In
addition this document of the \textbf{libcpuset} user library, the kernel cpuset
facility is documented in the kernel source file Documentation/cpusets.txt, and
Simon Derr has provided a document of cpusets at LINEBREAK
\href{http://www.bullopensource.org/cpuset/}{http://www.bullopensource.org/cpuset/}.

The user level bitmask library supports convenient manipulation of multi-word
bitmasks useful for CPUs and memory nodes. This bitmask library is required by
and designed to work with the cpuset library. The design notes of \textbf{libbitmask}
are available in a separate document, Bitmask{\_}Library.html or
Bitmask{\_}Library.pdf.

Unlike \texttt{sched{\_}setaffinity()} and \texttt{mbind()}, which are implemented as
additional kernel system calls, the primary kernel interface for accessing the
cpuset facility is \href{\#the-cpuset-file-system}{The Cpuset File System}, usually mounted at \texttt{/dev/cpuset}.
The cpuset library \textbf{libcpuset} provides convenient access to these facilities
from 'C' programs.

Cpusets extend the usefulness of the Linux 2.6 kernel mechanisms
\texttt{sched{\_}setaffinity()} for CPU placement, and \texttt{mbind()} and
\texttt{set{\_}mempolicy()} for memory placement. On smaller or dedicated use systems,
these other mechanisms are often sufficient. The \textbf{libcpuset} library provides
a convenient API to these other mechanisms that has the added advantage of being
robustly adapting to memory migration.

On larger NUMA systems, running more than one, performance critical,
job, it is necessary to be able to manage jobs in their entirety.
This includes providing a job with exclusive CPU and memory that no
other job can use and being able to list all tasks currently in a
cpuset.

You can use both these other placement mechanisms and cpusets together, using
the \href{\#advanced-cpuset-library-functions}{Advanced Cpuset Library Functions} to manage overall job placement, and using
the other mechanisms, perhaps via the \href{\#basic-cpuset-library-functions}{Basic Cpuset Library Functions} within each
given job to manage the details of thread and memory page placement.


%___________________________________________________________________________

\hypertarget{the-cpuset-file-system}{}
\pdfbookmark[1]{2.1~~~The Cpuset File System}{the-cpuset-file-system}
\subsection*{2.1~~~The Cpuset File System}

Cpusets are named, nested sets of CPUs and memory nodes. Each cpuset is
represented by a directory in the cpuset virtual file system, normally mounted
at \texttt{/dev/cpuset}.

New cpusets are created using the \texttt{mkdir} system call or shell command. The
properties of a cpuset, such as its flags, allowed CPUs and memory nodes, and
attached tasks, are queried and modified by reading or writing to the
appropriate file in that cpusets directory.

The state of each cpuset is represented by small text files in that cpusets
directory. These files may be read and written using traditional shell utilities
such as \texttt{cat(1)} and \texttt{echo(1)}, or using ordinary file access routines from
programmatic languages, such as \texttt{open(2)}, \texttt{read(2)}, \texttt{write(2)} and
\texttt{close(2)} from the 'C' library.

These per-cpuset files represent internal kernel state and do not have any
persistent image on disk. These files are automatically created when the cpuset
is created, as a result of the \texttt{mkdir} invocation. It is not allowed to add or
remove files from a cpuset directory.

Each of these per-cpuset files is listed and described below:
\begin{quote}
\begin{description}
%[visit_definition_list_item]
\item[tasks:] %[visit_definition]

List of the process ID's (PIDs) of the tasks in that cpuset. The list
is formatted as a series of ASCII decimal numbers, each followed by a
newline. A task may be added to a cpuset (removing it from the cpuset
previously containing it) by writing its PID to that cpusets \texttt{tasks}
file (with or without a trailing newline.)

Beware that only one PID may be written to the \texttt{tasks} file at a
time. If a string is written that contains more than one PID, all but
the first will be ignored.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[notify{\_}on{\_}release:] %[visit_definition]

Flag (0 or 1). If set (1), then that cpuset will receive special
handling whenever its last using task and last child cpuset goes away.
For more information, see the \titlereference{Notify On Release} section, below.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[cpus:] %[visit_definition]

List  of  CPUs on which tasks in that cpuset are allowed to execute.  See
\href{\#list-format}{List Format} below for a description of the format of \texttt{cpus}.

The CPUs allowed to a cpuset may be changed by writing a new list to  its
\texttt{cpus}  file.   Note  however, such a change does not take affect until the
PIDs of the tasks in the cpuset are rewritten to the cpusets \texttt{tasks}  file.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[cpu{\_}exclusive:] %[visit_definition]

Flag (0 or 1).  If set (1), then the cpuset has exclusive use of its CPUs
(no sibling or cousin cpuset may overlap CPUs).  By default this  is  off
(0).  Newly created cpusets also initially default this to off (0).

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[mems:] %[visit_definition]

List of memory nodes on which tasks in that cpuset are allowed to
allocate memory. See \href{\#list-format}{List Format} below for a description of the format
of \texttt{mems}.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[mem{\_}exclusive:] %[visit_definition]

Flag (0 or 1).  If set (1), then the cpuset has exclusive use of its memory
nodes (no sibling or cousin may overlap).  By  default  this  is  off
(0).  Newly created cpusets also initially default this to off (0).

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[memory{\_}migrate:] %[visit_definition]

Flag  (0  or  1).  If set (1), then memory migration is enabled.  For more information, see the \titlereference{Memory Migration} section, below.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[memory{\_}pressure:] %[visit_definition]

A measure of how much memory pressure the tasks in this cpuset are
causing. For more information, see the \titlereference{Memory Pressure} section, below. Always has value zero
(0) unless \texttt{memory{\_}pressure{\_}enabled} is enabled in the top cpuset.
This file is read-only.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[memory{\_}pressure{\_}enabled:] %[visit_definition]

Flag  (0  or  1).  This file is only present in the root cpuset, normally
\texttt{/dev/cpuset}.  If set (1), then \texttt{memory{\_}pressure} calculations  are  enabled
for all cpusets in the system.  For more information, see the \titlereference{Memory Pressure} section, below.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[memory{\_}spread{\_}page:] %[visit_definition]

Flag  (0  or  1).   If  set  (1), then the kernel page cache (file system
buffers) are uniformly spread across the cpuset.  For more information, see the \titlereference{Memory Spread} section, below.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[memory{\_}spread{\_}slab:] %[visit_definition]

Flag  (0  or  1).   If  set (1), then the kernel slab caches for file i/o
(directory and inode structures) are uniformly spread across the  cpuset.
For more information, see the \titlereference{Memory Spread} section, below.

%[depart_definition]
%[depart_definition_list_item]
\end{description}
\end{quote}

In addition one new file type is added to the \texttt{/proc} file system:
\begin{quote}
\begin{description}
%[visit_definition_list_item]
\item[/proc/{\textless}pid{\textgreater}/cpuset:] %[visit_definition]

For each task (pid), list its cpuset path, relative to the
root of the cpuset file system.  This file is read-only.

%[depart_definition]
%[depart_definition_list_item]
\end{description}
\end{quote}

Finally, the two control fields actually used by the kernel scheduler
and memory allocator to constrain scheduling and allocation to the
allowed CPUs are exposed as two more fields in the \texttt{status} file of
each task:
\begin{quote}

\textbf{/proc/{\textless}pid{\textgreater}/status}:
\begin{quote}
\begin{description}
%[visit_definition_list_item]
\item[\texttt{Cpus{\_}allowed}:] %[visit_definition]

bit vector of CPUs on which this task may be scheduled.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[\texttt{Mems{\_}allowed}:] %[visit_definition]

bit vector of memory nodes on which this task may obtain memory.

%[depart_definition]
%[depart_definition_list_item]
\end{description}
\end{quote}
\end{quote}

There are several reasons why a tasks \texttt{Cpus{\_}allowed} and \texttt{Mems{\_}allowed} values
may differ from the \texttt{cpus} and \texttt{mems} that are allowed in its current cpuset, as follows:
\begin{quote}
\newcounter{listcnt1}
\begin{list}{\Alph{listcnt1}.}
{
\usecounter{listcnt1}
\setlength{\rightmargin}{\leftmargin}
}
\item {} 
A task might use \texttt{sched{\_}setaffinity}, \texttt{mbind} or \texttt{set{\_}mempolicy}
to restrain its placement to less than its cpuset.

\item {} 
Various temporary changes to \texttt{Cpus{\_}allowed} are done by
kernel internal code.

\item {} 
Attaching a task to a cpuset doesn't change its \texttt{Mems{\_}allowed}
until the next time that task needs kernel memory.

\item {} 
Changing a cpusets \texttt{cpus} doesn't change the \texttt{Cpus{\_}allowed} of
the tasks attached to it until those tasks are reattached
to that cpuset (to avoid a hook in the hotpath scheduler
code in the kernel).

\item {} 
If hotplug is used to remove all the CPUs, or all the memory nodes,
in a cpuset, then the tasks attached to that cpuset will have their
\texttt{Cpus{\_}allowed} or \texttt{Mems{\_}allowed} altered to the CPUs or memory
nodes of the closest ancestor to that cpuset that is not empty.

\end{list}
\end{quote}

Beware of items D and E, above. Due to item D, user space action is required to
update a tasks \texttt{Cpus{\_}allowed} after changing its cpuset. Use the routine
\href{\#cpuset-reattach}{cpuset{\_}reattach} to perform this update after a changing the \texttt{cpus} allowed to
a cpuset.

Due to item E, the confines of a cpuset can be \textbf{violated} after a hotplug
removal that empties a cpuset. To avoid having a cpuset without CPU or memory
resources, update your system's cpuset configuration to reflect the new hardware
configuration. The kernel prefers misplacing a task, over starving a task of
essential compute resources.

There is one other condition under which the confines of a cpuset may be
violated. A few kernel critical internal memory allocation requests, marked
\texttt{GFP{\_}ATOMIC}, must be satisfied immediately. The kernel may drop some request
or malfunction if one of these allocations fail. If such a request cannot be
satisfied within the current tasks cpuset, then the kernel relaxes the cpuset,
and looks for memory anywhere it can find it. It's better to violate the cpuset
than stress the kernel.

New cpusets are created using \texttt{mkdir} (at the shell or in C).
Old ones are removed using \texttt{rmdir}.  The above files are accessed
using \texttt{read(2)} and \texttt{write(2)} system calls, or shell commands such
as \texttt{cat(1)} and \texttt{echo(1)}.

The CPUs and memory nodes in a given cpuset are always a subset
of its parent.  The root cpuset has all possible CPUs and memory
nodes in the system.  A cpuset may be exclusive (cpu or memory)
only if its parent is similarly exclusive.

Each task has a pointer to a cpuset.  Multiple tasks may reference
the same cpuset.  Requests by a task, using the \texttt{sched{\_}setaffinity(2)}
system call to include CPUs in its CPU affinity mask, and using the
\texttt{mbind(2)} and \texttt{set{\_}mempolicy(2)} system calls to include memory nodes
in its memory policy, are both filtered through that tasks cpuset,
filtering out any CPUs or memory nodes not in that cpuset.  The
scheduler will not schedule a task on a CPU that is not allowed in
its cpus{\_}allowed vector, and the kernel page allocator will not
allocate a page on a node that is not allowed in the requesting tasks
mems{\_}allowed vector.

If a cpuset is cpu or mem exclusive, no other cpuset, other than a direct
ancestor or descendant, may share any of the same CPUs or memory nodes.

User level code may create and destroy cpusets by name in the cpuset
virtual file system, manage the attributes and permissions of these
cpusets and which CPUs and memory nodes are assigned to each cpuset,
specify and query to which cpuset a task is assigned, and list the
task pids assigned to a cpuset.

Cpuset names are limited in length by the kernel's VFS implementation.
No single component of a cpuset name may exceed 255 characters, and
the full pathname of a cpuset including the \texttt{/dev/cpuset} mount
point may not exceed 4095 characters in length.


%___________________________________________________________________________

\hypertarget{exclusive-cpusets}{}
\pdfbookmark[1]{2.2~~~Exclusive Cpusets}{exclusive-cpusets}
\subsection*{2.2~~~Exclusive Cpusets}

If  a  cpuset  is  marked \texttt{cpu{\_}exclusive} or \texttt{mem{\_}exclusive}, no other cpuset, other
than a direct ancestor or descendant, may share any of the same CPUs  or  memory
nodes.

A cpuset that is \texttt{cpu{\_}exclusive} has a scheduler (sched) domain associated
with it. The sched domain consists of all CPUs in the current cpuset that are
not part of any exclusive child cpusets. This ensures that the scheduler load
balancing code only balances against the CPUs that are in the sched domain as
defined above and not all of the CPUs in the system. This removes any overhead
due to load balancing code trying to pull tasks outside of the \texttt{cpu{\_}exclusive}
cpuset only to be prevented by the  \texttt{Cpus{\_}allowed} mask of the task.

A cpuset that is \texttt{mem{\_}exclusive} restricts kernel allocations for page, buffer, and
other data commonly shared by the kernel across multiple  users.   All  cpusets,
whether  \texttt{mem{\_}exclusive}  or  not,  restrict allocations of memory for user space.
This enables configuring a system so that several  independent  jobs  can  share
common  kernel  data,  such as file system pages, while isolating each jobs user
allocation in its own cpuset.  To  do  this,  construct  a  large  \texttt{mem{\_}exclusive}
cpuset  to hold all the jobs, and construct child, non-mem{\_}exclusive cpusets for
each individual job.  Only a small amount of  typical  kernel  memory,  such  as
requests  from  interrupt  handlers,  is  allowed  to  be  taken  outside even a
\texttt{mem{\_}exclusive} cpuset.


%___________________________________________________________________________

\hypertarget{notify-on-release}{}
\pdfbookmark[1]{2.3~~~Notify On Release}{notify-on-release}
\subsection*{2.3~~~Notify On Release}

If the \texttt{notify{\_}on{\_}release} flag is enabled (1) in a cpuset, then whenever the last
task  in the cpuset leaves (exits or attaches to some other cpuset) and the last
child cpuset of that cpuset  is  removed, the  kernel  runs  the  command
\texttt{/sbin/cpuset{\_}release{\_}agent},  supplying the pathname (relative to the mount point
of the cpuset file system) of the  abandoned  cpuset.   This  enables  automatic
removal of abandoned cpusets.

The default value of \texttt{notify{\_}on{\_}release} in the root cpuset at system boot is
disabled (0). The default value of other cpusets at creation is the current
value of their parents \texttt{notify{\_}on{\_}release} setting.

The  command  \texttt{/sbin/cpuset{\_}release{\_}agent}  is invoked, with the name (\texttt{/dev/cpuset}
relative path) of that cpuset in \texttt{argv{[}1]}.  This supports  automatic  cleanup  of
abandoned cpusets.

The usual contents of the command \texttt{/sbin/cpuset{\_}release{\_}agent} is simply the shell
script:
\begin{quote}{\ttfamily \raggedright \noindent
{\#}!/bin/sh~\\
rmdir~/dev/cpuset/{\$}1
}\end{quote}

By default \texttt{notify{\_}on{\_}release} is off (0).  Newly created  cpusets  inherit  their
\texttt{notify{\_}on{\_}release} setting from their parent cpuset.

As  with  other  flag values below, this flag can be changed by writing an ASCII
number 0 or 1 (with optional trailing newline) into the file, to  clear  or  set
the flag, respectively.


%___________________________________________________________________________

\hypertarget{memory-pressure}{}
\pdfbookmark[1]{2.4~~~Memory Pressure}{memory-pressure}
\subsection*{2.4~~~Memory Pressure}

The  \texttt{memory{\_}pressure} of a cpuset provides a simple per-cpuset metric of the rate
that the tasks in a cpuset are attempting to free up in use memory on the  nodes
of the cpuset to satisfy additional memory requests.

This  enables  batch schedulers  monitoring  jobs running in dedicated cpusets to
efficiently detect what level of memory pressure that job is causing.

This is useful both on tightly managed systems running a wide mix of submitted
jobs, which may choose to terminate or re-prioritize jobs that are trying to use
more memory than allowed on the nodes assigned them, and with tightly coupled,
long running, massively parallel scientific computing jobs that will
dramatically fail to meet required performance goals if they start to use more
memory than allowed to them.

This mechanism provides a very economical way for the batch scheduler to monitor a
cpuset for signs of memory pressure.  It's up to the batch scheduler or other user
code to decide what to do about it and take action.

If the \texttt{memory{\_}pressure{\_}enabled} flag in the top cpuset is not set (0), then
the kernel does not compute this filter, and the per-cpuset files
\texttt{memory{\_}pressure} always contain the value zero (0).

If the \texttt{memory{\_}pressure{\_}enabled} flag in the top cpuset is set (1), then the
kernel computes this filter for each cpuset in the system, and the
memory{\_}pressure file for each cpuset reflects the recent rate of
such low memory page allocation attempts by tasks in said cpuset.

Reading the \texttt{memory{\_}pressure} file of a cpuset is very efficient.
The expectation is that batch schedulers can poll these files and
detect jobs that are causing memory stress, so that action can be
taken to avoid impacting the rest of the system with a job that
is trying to aggressively exceed its allowed memory.

\emph{Note well}: unless enabled by setting \texttt{memory{\_}pressure{\_}enabled} in the top cpuset,
\texttt{memory{\_}pressure} is not computed for any cpuset, and always reads a value of
zero.

Why a per-cpuset, running average:
\begin{quote}

Because this meter is per-cpuset, rather than per-task or memory region, the system
load imposed by a batch scheduler monitoring this metric is sharply
reduced on large systems, because a scan of the system-wide tasklist can
be avoided on each set of queries.

Because this meter is a  running  average,  instead  of  an  accumulating
counter, a batch scheduler can detect memory pressure with a single read,
instead of having to read and accumulate results for a period of time.

Because this meter is per-cpuset rather than per-task or  memory region,  the  batch
scheduler  can  obtain  the key information, memory pressure in a cpuset,
with a single read, rather than having to query  and  accumulate  results
over all the (dynamically changing) set of tasks in the cpuset.
\end{quote}

A per-cpuset simple digital filter is kept within the kernel, and updated by any
task attached to that cpuset, if it enters the synchronous (direct) page reclaim
code.

The per-cpuset \texttt{memory{\_}pressure} file provides an integer number representing
the recent (half-life of 10 seconds) rate of direct page reclaims caused by the
tasks in the cpuset, in units of reclaims attempted per second, times 1000.

The kernel computes this value using a single-pole low-pass recursive (IIR)
digital filter coded with 32 bit integer arithmetic.  The value decays
at an exponential rate.

Given the simple 32 bit integer arithmetic used in the kernel to compute this value,
this meter works best for reporting page reclaim rates between one per millisecond (msec)
and one per 32 (approx) seconds.  At constant rates faster than one per msec it maxes
out at values just under 1,000,000.  At constant rates between one per msec, and
one per second it will stabilize to a value N*1000, where N is the rate of events
per second.  At constant rates between one per second and one per 32 seconds,
it will be choppy, moving up on the seconds that have an event, and then decaying
until the next event.  At rates slower than about one in 32 seconds, it decays all
the way back to zero between each event.


%___________________________________________________________________________

\hypertarget{memory-spread}{}
\pdfbookmark[1]{2.5~~~Memory Spread}{memory-spread}
\subsection*{2.5~~~Memory Spread}

There are two Boolean flag files per cpuset that control where the kernel
allocates pages for the file system buffers and related in kernel data
structures. They are called \texttt{memory{\_}spread{\_}page} and \texttt{memory{\_}spread{\_}slab}.

If the per-cpuset Boolean flag file \texttt{memory{\_}spread{\_}page} is set, the
kernel will spread the file system buffers (page cache) evenly over all the
nodes that the faulting task is allowed to use, instead of preferring to put
those pages on the node where the task is running.

If the per-cpuset Boolean flag file \texttt{memory{\_}spread{\_}slab} is set, the
kernel will spread some file system related slab caches, such as for inodes and
directory entries, evenly over all the nodes that the faulting task is allowed to
use; instead of preferring to put those pages on the node where the task is
running.

The setting of these flags does not affect anonymous data segment or stack
segment pages of a task.

By default, both kinds of memory spreading are off, and memory pages are
allocated on the node local to where the task is running, except perhaps as
modified by the tasks NUMA mempolicy or cpuset configuration, as long as
sufficient free memory pages are available.

When new cpusets are created, they inherit the memory spread settings  of  their
parent.

Setting memory spreading causes allocations for the affected page or slab caches
to ignore the tasks NUMA mempolicy and be spread instead.    Tasks using mbind()
or  set{\_}mempolicy()  calls to set NUMA mempolicies will not notice any change in
these calls as a result of their containing tasks memory  spread  settings.   If
memory spreading is turned off,  the currently specified NUMA mempolicy once
again applies to memory page allocations.

Both \texttt{memory{\_}spread{\_}page} and \texttt{memory{\_}spread{\_}slab} are Boolean flag files.
By default they contain ``0'', meaning that the feature is off for that cpuset. If
a ``1'' is written to that file,  that turns the named feature on.

This memory placement policy is also known (in other contexts) as round-robin or
interleave.

This  policy  can  provide  substantial improvements for jobs that need to place
thread local data on the corresponding node, but that need to access large  file
system  data  sets  that  need to be spread across the several nodes in the jobs
cpuset in order to fit.  Without this policy, especially  for  jobs  that  might
have  one thread reading in the data set, the memory allocation across the nodes
in the jobs cpuset can become very uneven.


%___________________________________________________________________________

\hypertarget{memory-migration}{}
\pdfbookmark[1]{2.6~~~Memory Migration}{memory-migration}
\subsection*{2.6~~~Memory Migration}

Normally, under the default setting (disabled) of \texttt{memory{\_}migrate}, once a
page is allocated (given a physical page of main memory)  that page stays on
whatever node it was allocated, as long as it remains allocated.  If the
cpuset memory placement policy \texttt{mems} subsequently changes, currently
allocated pages are not moved.  If pages are swapped out to disk and back,
then on return to main memory, they may be allocated on different nodes,
depending on the cpuset \texttt{mems} setting in affect at the time the page
is swapped back in.

When memory migration is enabled in a cpuset, if the \texttt{mems} setting of the
cpuset is changed, then any memory page in use by any task in the cpuset that is
on a memory node no longer allowed will be migrated to a memory node that is
allowed.

Also if a task is moved into a cpuset with \texttt{memory{\_}migrate} enabled, any
memory pages it uses that were on memory nodes allowed in its previous cpuset,
but which are not allowed in its new cpuset, will be migrated to a memory node
allowed in the new cpuset.

The relative placement of a migrated page within the cpuset is preserved during
these migration operations if possible. For example, if the page was on the
second valid node of the prior cpuset, the page will be placed on the second
valid node of the new cpuset, if possible.

In order to maintain the cpuset relative position of pages, even pages on memory
nodes allowed in both the old and new cpusets may be migrated.  For example, if
\texttt{memory{\_}migrate} is enabled in a cpuset, and that cpusets \texttt{mems} file is
written, changing it from say memory nodes ``4-7'', to memory nodes ``5-8'', then
the following page migrations will be done, in order, for all pages in the
address space of tasks in that cpuset:
\begin{quote}{\ttfamily \raggedright \noindent
First,~migrate~pages~on~node~7~to~node~8~\\
Second,~migrate~pages~on~node~6~to~node~7~\\
Third,~migrate~pages~on~node~5~to~node~6~\\
Fourth,~migrate~pages~on~node~4~to~node~5
}\end{quote}

In this example, pages on any memory node other than ``4-7'' will not be migrated.
The order in which nodes are handled in a migration is intentionally chosen so as
to avoid migrating memory \titlereference{to} a node until any migrations \titlereference{from} that node have
first been accomplished.


%___________________________________________________________________________

\hypertarget{mask-format}{}
\pdfbookmark[1]{2.7~~~Mask Format}{mask-format}
\subsection*{2.7~~~Mask Format}

The \href{\#mask-format}{Mask Format} is used to represent CPU and memory node bitmasks in the
\textbf{/proc/{\textless}pid{\textgreater}/status} file.

It is hexadecimal, using ASCII characters ``0'' - ``9'' and ``a'' - ``f''. This format
displays each 32-bit word in hex (zero filled), and for masks longer than one
word, uses a comma separator between words. Words are displayed in big-endian
order most significant first. And hex digits within a word are also in
big-endian order.

The number of 32-bit words displayed is the minimum number needed to display all
bits of the bitmask, based on the size of the bitmask.

Examples of the \href{\#mask-format}{Mask Format}:
\begin{quote}
\begin{quote}{\ttfamily \raggedright \noindent
00000001~~~~~~~~~~~~~~~~~~~~~~~~{\#}~just~bit~0~set~\\
80000000,00000000,00000000~~~~~~{\#}~just~bit~95~set~\\
00000001,00000000,00000000~~~~~~{\#}~just~bit~64~set~\\
000000ff,00000000~~~~~~~~~~~~~~~{\#}~bits~32-39~set~\\
00000000,000E3862~~~~~~~~~~~~~~~{\#}~bits~1,5,6,11-13,17-19~set
}\end{quote}
\end{quote}

A mask with bits 0, 1, 2, 4, 8, 16, 32 and 64 set displays as
``00000001,00000001,00010117''.  The first ``1'' is for bit 64, the second
for bit 32, the third for bit 16, the fourth for bit 8, the fifth for
bit 4, and the ``7'' is for bits 2, 1 and 0.


%___________________________________________________________________________

\hypertarget{list-format}{}
\pdfbookmark[1]{2.8~~~List Format}{list-format}
\subsection*{2.8~~~List Format}

The \href{\#list-format}{List Format} is used to represent CPU and memory node
bitmasks (sets of CPU and memory node numbers) in the
\texttt{/dev/cpuset} file system.

It is a comma separated list  of  CPU  or  memory
node  numbers  and ranges of numbers, in ASCII decimal.

Examples of the \href{\#list-format}{List Format}:
\begin{quote}
\begin{quote}{\ttfamily \raggedright \noindent
0-4,9~~~~~~~~~~~{\#}~bits~0,~1,~2,~3,~4,~and~9~set~\\
0-3,7,12-15~~~~~{\#}~bits~0,~1,~2,~3,~7,~12,~13,~14,~and~15~set
}\end{quote}
\end{quote}


%___________________________________________________________________________

\hypertarget{using-cpusets-at-the-shell-prompt}{}
\pdfbookmark[0]{3~~~Using Cpusets at the Shell Prompt}{using-cpusets-at-the-shell-prompt}
\section*{3~~~Using Cpusets at the Shell Prompt}

There are multiple ways to use cpusets, including:
\begin{quote}
\begin{itemize}
\item {} 
They can be queried and changed from a shell prompt,
using such command line utilities as \texttt{echo}, \texttt{cat},
\texttt{mkdir} and \texttt{ls}.

\item {} 
They can be queried and changed via the \textbf{libcpuset}
'C' programming API.  The primary emphasis of this document
is on the 'C' API.

\end{itemize}
\end{quote}

This section describes the use of cpusets using shell commands.

One convenient way to learn how cpusets work is to experiment
with them at the shell prompt, before doing extensive 'C'
coding.

Note that there is one significant difference between these two interfaces.

Modifying the CPUs in a cpuset the shell prompt requires an additional step, due
to intentional limitations in the kernel support for cpusets. The
\href{\#cpuset-reattach}{cpuset{\_}reattach} routine can be used to perform this step when using
\textbf{libcpuset}.  The extra step consists of writing the pid of each task attached
to that cpuset back into the cpusets \texttt{tasks} file:
\begin{quote}

In order to minimize the impact of cpusets on critical kernel
code, such as the scheduler, and due to the fact that the kernel
does not support one task updating the memory placement of another
task directly, the impact on a task of changing its cpuset CPU
or memory node placement, or of changing to which cpuset a task
is attached, is subtle.

If a cpuset has its memory nodes modified, then for each task attached
to that cpuset, the next time that the kernel attempts to allocate
a page of memory for that task, the kernel will notice the change
in the tasks cpuset, and update its per-task memory placement to
remain within the new cpusets memory placement.  If the task was using
mempolicy MPOL{\_}BIND, and the nodes to which it was bound overlap with
its new cpuset, then the task will continue to use whatever subset
of MPOL{\_}BIND nodes are still allowed in the new cpuset.  If the task
was using MPOL{\_}BIND and now none of its MPOL{\_}BIND nodes are allowed
in the new cpuset, then the task will be essentially treated as if it
was MPOL{\_}BIND bound to the new cpuset (even though its NUMA placement,
as queried by get{\_}mempolicy(), doesn't change).  If a task is moved
from one cpuset to another, then the kernel will adjust the tasks
memory placement, as above, the next time that the kernel attempts
to allocate a page of memory for that task.

If a cpuset has its CPUs modified, then each task using that cpuset does
{\_}not{\_} change its behavior automatically. In order to minimize the impact on
the critical scheduling code in the kernel, tasks will continue to use their
prior CPU placement until they are rebound to their cpuset, by rewriting
their pid to the 'tasks' file of their cpuset. If a task had been bound to
some subset of its cpuset using the sched{\_}setaffinity() call, the effect of
this is lost on the rebinding. The rebound tasks \texttt{cpus{\_}allowed} is set to
include all \texttt{cpus} in the tasks new cpuset. If a task is moved from one
cpuset to another, its CPU placement is updated in the same way as if the
tasks pid is rewritten to the 'tasks' file of its current cpuset.

In summary, the memory placement of a task whose cpuset is changed is
automatically updated by the kernel, on the next allocation of a page for
that task, but the processor placement is not updated, until that tasks pid
is rewritten to the 'tasks' file of its cpuset. The delay in rebinding a
tasks memory placement is necessary because the kernel does not support one
task changing another tasks memory placement. The added user level step in
rebinding a tasks CPU placement is necessary to avoid impacting the
scheduler code in the kernel with a check for changes in a tasks processor
placement.
\end{quote}

To create a new cpuset and attach the current command shell to it, the steps are:
\begin{quote}
\newcounter{listcnt2}
\begin{list}{\arabic{listcnt2})}
{
\usecounter{listcnt2}
\setlength{\rightmargin}{\leftmargin}
}
\item {} 
mkdir /dev/cpuset (if not already done)

\item {} 
mount -t cpuset cpuset /dev/cpuset (if not already done)

\item {} 
Create the new cpuset using \texttt{mkdir(1)}.

\item {} 
Assign CPUs and memory nodes to the new cpuset.

\item {} 
Attach the shell to the new cpuset.

\end{list}
\end{quote}

For example, the following sequence of commands will setup a cpuset
named ``Charlie'', containing just CPUs 2 and 3, and memory node 1,
and then attach the current shell to that cpuset:
\begin{quote}
\begin{quote}{\ttfamily \raggedright \noindent
mkdir~/dev/cpuset~\\
mount~-t~cpuset~cpuset~/dev/cpuset~\\
cd~/dev/cpuset~\\
mkdir~Charlie~\\
cd~Charlie~\\
/bin/echo~2-3~>~cpus~\\
/bin/echo~1~>~mems~\\
/bin/echo~{\$}{\$}~>~tasks~\\
{\#}~The~current~shell~is~now~running~in~cpuset~Charlie~\\
{\#}~The~next~line~should~display~'/Charlie'~\\
cat~/proc/self/cpuset
}\end{quote}
\end{quote}

Migrating a job (the set of tasks attached to a cpuset) to different CPUs and
memory nodes in the system, including moving the memory pages currently
allocated to that job, can be done as follows.  Lets say you want to move the job
in cpuset \emph{alpha} (CPUs 4-7 and memory nodes 2-3) to a new cpuset \emph{beta} (CPUs
16-19 and memory nodes 8-9).
\begin{quote}
\newcounter{listcnt3}
\begin{list}{\arabic{listcnt3})}
{
\usecounter{listcnt3}
\setlength{\rightmargin}{\leftmargin}
}
\item {} 
First create the new cpuset \emph{beta}.

\item {} 
Then allow CPUs 16-19 and memory nodes 8-9 in \emph{beta}.

\item {} 
Then enable \texttt{memory{\_}migration} in \emph{beta}.

\item {} 
Then move each task from \emph{alpha} to \emph{beta}.

\end{list}
\end{quote}

The following sequence of commands accomplishes this:
\begin{quote}{\ttfamily \raggedright \noindent
cd~/dev/cpuset~\\
mkdir~beta~\\
cd~beta~\\
/bin/echo~16-19~>~cpus~\\
/bin/echo~8-9~>~mems~\\
/bin/echo~1~>~memory{\_}migrate~\\
while~read~i;~do~/bin/echo~{\$}i;~done~<~../alpha/tasks~>~tasks
}\end{quote}

The  above  should move any tasks in \emph{alpha} to \emph{beta}, and any memory held by these
tasks on memory nodes 2-3 to memory nodes 8-9, respectively.

Notice that the last step of the above sequence did \textbf{not} do:
\begin{quote}{\ttfamily \raggedright \noindent
cp~../alpha/tasks~tasks~~~~~{\#}~Doesn't~work~(ignores~all~but~first~task)
}\end{quote}

The \textbf{while} loop, rather than the seemingly easier use of the \texttt{cp(1)}
command, is necessary because only one task PID at a time may be written to the
\texttt{tasks} file.

The same affect (writing one pid at a time) as the \textbf{while} loop can be
accomplished more efficiently, in fewer keystrokes and in syntax that works in
any shell, but alas more obscurely, by using the \textbf{sed -u {[}unbuffered]} option:
\begin{quote}{\ttfamily \raggedright \noindent
sed~-un~p~<~../alpha/tasks~>~tasks
}\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-programming-model}{}
\pdfbookmark[0]{4~~~Cpuset Programming Model}{cpuset-programming-model}
\section*{4~~~Cpuset Programming Model}

The \textbf{libcpuset} programming model for cpusets provides a hierarchical
cpuset name space that integrates smoothly with Linux 2.6 kernel support for
simple processor and memory placement.  As systems become larger, with more
complex memory, processor and bus architectures, the hierarchical cpuset
model for managing processor and memory resources will become increasingly
important.

The cpuset name space remains visible to all tasks on a system. Once created, a
cpuset remains in existence until it is deleted or until the system is rebooted,
even if no tasks are currently running in that cpuset.

The key properties of a cpuset are its pathname, the list of which CPUs and
memory nodes it contains, and whether the cpuset has exclusive rights to these
resources.

Every task (process) in the system is attached to (running inside) a cpuset.
Tasks inherit their parents cpuset attachment when forked. This binding of task
to a cpuset can subsequently be changed, either by the task itself, or
externally from another task, given sufficient authority.

Tasks have their CPU and memory placement constrained to whatever their
containing cpuset allows. A cpuset may have exclusive rights to its CPUs and
memory, which provides certain guarantees that other cpusets will not overlap.

At system boot, a top level root cpuset is created, which includes all CPUs and
memory nodes on the system. The usual mount point of the cpuset file system, and
hence the usual file system path to this root cpuset, is \texttt{/dev/cpuset}.

Changing the cpuset binding of a task does not by default move the memory the
tasks might have currently allocated, even if that memory is on memory nodes no
longer allowed in the tasks cpuset. On kernels that support such memory
migration, use the \textbf{{[}optional]} \emph{cpuset{\_}migrate} function to move allocated
memory as well.

To create a cpuset from 'C' code, one obtains a handle to a new \texttt{struct
cpuset}, sets the desired attributes via that handle, and issues a
\texttt{cpuset{\_}create()} to actually create the desired cpuset and bind it to the
specified name. One can also list by name what cpusets exist, query and modify
their properties, move tasks between cpusets, list what tasks are currently
attached to a cpuset, and delete cpusets.

The \texttt{cpuset{\_}alloc()} call applies a hidden \emph{undefined} mark to each attribute
of the allocated struct cpuset. Calls to the various \texttt{cpuset{\_}set*()} routines
mark the attribute being set as \emph{defined}. Calls to \texttt{cpuset{\_}create()} and
\texttt{cpuset{\_}modify()} only set the attributes of the cpuset marked \emph{defined}. This
is primarily noticable when creating a cpuset. Code in the kernel sets some
attributes of new cpusets, such as \texttt{memory{\_}spread{\_}page},
\texttt{memory{\_}spread{\_}slab} and \texttt{notify{\_}on{\_}release}, by default to the value
inherited from their parent. Unless the application using \texttt{libcpuset}
explicitly overrides the setting of these attributes in the struct cpuset,
between the calls to \texttt{cpuset{\_}alloc()} and \texttt{cpuset{\_}create()}, the kernel
default settings will prevail. These hidden marks have no noticeable affect when
modifying an existing cpuset using the sequence of calls \texttt{cpuset{\_}alloc()},
\texttt{cpuset{\_}query()}, and \texttt{cpuset{\_}modify()}, because the \texttt{cpuset{\_}query()} call
sets all attributes and marks them \emph{defined}, while reading the attributes from
the cpuset.

The names of cpusets in this 'C' library are either relative to the
root cpuset mount point (typically \texttt{/dev/cpuset}) if the name starts
with a slash '/' character or else relative to the current tasks cpuset.

Cpusets can be renamed using the \texttt{rename(2)} system call.  The per-cpuset
files within each cpuset directory cannot be renamed, and the \texttt{rename(2)}
system call cannot be used to modify the cpuset hierarchy. You cannot change the
parent cpuset of a cpuset using \texttt{rename(2)}.

Despite its name, the \texttt{pid} parameter to various \texttt{libcpuset} routines is actually
a thread id, and each thread in a threaded group can be attached to a different
cpuset.  The value returned from a call to \texttt{gettid(2)} can be passed in the
argument \texttt{pid}.


%___________________________________________________________________________

\hypertarget{permission-model}{}
\pdfbookmark[1]{4.1~~~Permission Model}{permission-model}
\subsection*{4.1~~~Permission Model}

Cpusets have a permission structure which determines which users have rights to
query, modify and attach to any given cpuset. The permissions of a cpuset are
determined by the permissions of the special files and directories in the cpuset
file system.  The cpuset filesystem is normally mounted at \texttt{/dev/cpuset}.

The directory representing each cpuset, and the special per-cpuset files in each
such directory, both have traditional \texttt{Unix} hierarchical file system
permissions for read, write and execute (or search), for each of the owning
user, the owning group, and all others.

For instance, a task can put itself in some other cpuset (than its current one)
if it can write the tasks file for that cpuset (requires execute permission on
the encompassing directories and write permission on that tasks file).

Because the cpuset file system is not persistent, changes in permissions, and
even the existence of cpusets other than the root cpuset, are not preserved
across system reboots.

An additional constraint is applied to requests to place some other task in a
cpuset. One task may not attach another to a cpuset unless it would have
permission to send that task a signal.

A task may create a child cpuset if it can access and write the parent cpuset
directory. It can modify the CPUs or memory nodes in a cpuset if it can access
that cpusets directory (execute permissions on the encompassing directories) and
write the corresponding \texttt{cpus} or \texttt{mems} file.

It should be noted, however, that changes to the CPUs of a cpuset do not apply
to any task in that cpuset until the task is reattached to that cpuset. If a
task can write the cpus file, it should also be able to write the tasks file and
might be expected to have permission to reattach the tasks therein (equivalent
to permission to send them a signal).

Some utilities and libraries read the cpuset special files of the cpuset in
which the calling task is placed, and may be hindered if a task is placed in a
cpuset that is not owned by the task's user-id, unless special care is taken to
allow that task continued read access to its own cpuset special files (and
search access to the directory containing them).  For example, the Message
Passing Toolkit's MPI library's placement functionality activated through the
use of MPI{\_}DSM{\_}DISTRIBUTE will fail to place ranks correctly if it cannot read
the cpuset special files through insufficient permissions on the cpuset
directory or the special files within it.

There is one minor difference between the manner in which cpuset path
permissions are evaluated by \texttt{libcpuset} and the manner in which file system
operation permissions are evaluated by direct system calls.  System calls that
operate on file pathnames, such as the \texttt{open(2)} system call, rely on direct
kernel support for a tasks current directory. Therefore such calls can
successfully operate on files in or below a tasks current directory even if the
task lacks search permission on some ancestor directory. Calls in \texttt{libcpuset}
that operate on cpuset pathnames, such as the \texttt{cpuset{\_}query()} call, rely on
\texttt{libcpuset} internal conversion of all cpuset pathnames to full, root-based
paths, so cannot successfully operate on a cpuset unless the task has search
permission on all ancestor directories, starting with the usual cpuset mount
point (\texttt{/dev/cpuset}.)


%___________________________________________________________________________

\hypertarget{other-placement-mechanisms}{}
\pdfbookmark[1]{4.2~~~Other Placement Mechanisms}{other-placement-mechanisms}
\subsection*{4.2~~~Other Placement Mechanisms}

The Linux 2.6 kernel supports additional processor and memory placement mechanisms.
\begin{itemize}
\item {} 
The \texttt{sched{\_}setaffinity(2)} and \texttt{sched{\_}getaffinity(2)} system calls set and get
a process's CPU affinity mask, which determines the set of CPUs on which it is
eligible to run. The \texttt{taskset(1)} command provides a command line utility
for manipulating a process's CPU affinity mask using these system calls. Only
CPUs within a tasks cpuset are allowed, but CPUs are numbered using system
wide CPU numbers, not cpuset relative numbers.

\item {} 
The \texttt{set{\_}mempolicy(2)}, and \texttt{get{\_}mempolicy(2)} system calls set and get
a tasks NUMA memory policy for a process and its children, which defines on
which nodes memory is allocated for the process. The \texttt{numactl(8)} command
provides a command line utility and the \texttt{libnuma} library provides a 'C'
API for manipulating a process's NUMA memory policy using these system calls.
Only memory nodes within a tasks cpuset are allowed, but nodes are numbered
using system wide node numbers, not cpuset relative numbers.

\item {} 
The \texttt{mbind(2)} system call sets the NUMA memory policy for the pages in a specific
range of a tasks virtual address space.

\end{itemize}

Cpusets are designed to interact cleanly with these other mechanisms, to support
for example having a batch scheduler use cpusets to control the CPU and memory
placement of various jobs, while within each job, these other mechanisms are
used to manage placement in more detail. It is possible for a batch scheduler to
change a jobs cpuset placement while preserving the internal CPU affinity and
NUMA memory placement policy.

The CPU and memory node placement constraints imposed by cpusets always
constrain those of these other mechanisms. You can use these other mechanisms to
reduce further the set of CPUs or memory nodes allowed to a task by cpusets, but
you can not use these other mechanisms to escape a tasks cpuset confinement.

Calls to \texttt{sched{\_}setaffinity(2)} to modify a tasks CPU affinity automatically
mask off CPUs that are not allowed by the affected tasks cpuset. If that results
in all the requested CPUs being masked off, then the call fails with errno set
to \texttt{EINVAL}. If some of the requested CPUs are allowed by the tasks cpuset,
then the call proceeds as if only the allowed CPUs were requested, silently
ignoring the other, unallowed CPUs. If a task is moved to a different cpuset, or
if the \texttt{cpus} of a cpuset are changed, then the CPU affinity of the affected
task or tasks is lost. If a batch scheduler needs to preserve the CPU affinity of
the tasks in a job being moved, it should use the \texttt{sched{\_}getaffinity(2)} and
\texttt{sched{\_}setaffinity(2)} calls to save and restore each affected tasks CPU
affinity across the move, relative to the cpuset. The \textbf{cpu{\_}set{\_}t} mask data
type supported by the 'C' library for use with the CPU affinity calls is
different then the \textbf{libbitmask} bitmasks used by \textbf{libcpuset}, so some coding
is required to convert between the two, in order to calculate and preserve
cpuset relative CPU affinity.

Similar to CPU affinity, calls to modify a tasks NUMA memory policy silently
mask off requested memory nodes outside the tasks allowed cpuset, and will fail
if that results in requesting an empty set of memory nodes. Unlike CPU affinity,
the NUMA memory policy system calls do not support one task querying or
modifying another tasks policy. So the kernel automatically handles preserving
cpuset relative NUMA memory policy when either a task is attached to a different
cpuset, or a cpusets \texttt{mems} setting is changed. If the old and new \texttt{mems}
sets have the same size, then the cpuset relative offset of affected NUMA memory
policies is preserved. If the new \texttt{mems} is smaller, then the old \texttt{mems}
relative offsets are folded onto the new \texttt{mems}, modulo the size of the new
\texttt{mems}. If the new \texttt{mems} is larger, then just the first \texttt{N} nodes are used,
where \texttt{N} is the size of the old \texttt{mems}.


%___________________________________________________________________________

\hypertarget{cpuset-aware-thread-pinning}{}
\pdfbookmark[1]{4.3~~~Cpuset Aware Thread Pinning}{cpuset-aware-thread-pinning}
\subsection*{4.3~~~Cpuset Aware Thread Pinning}

If a job intends to use the \href{\#other-placement-mechanisms}{Other Placement Mechanisms} described above, then
that job cannot be \emph{guaranteed} safe operation under the control of a batch
scheduler if that job might be migrated to different CPUs or memory nodes. This is
because these \href{\#other-placement-mechanisms}{Other Placement Mechanisms} use system wide numbering of CPUs
and memory nodes, not cpuset relative numbering, and the job might be migrated
without its knowledge while it is trying to adjust its placement.

That is, between the point where such an application computes the CPU or memory
node on which it wants to place a thread, and the point where it issues the
\texttt{sched{\_}setaffinity(2)}, \texttt{mbind(2)} or \texttt{set{\_}mempolicy(2)} call to direct
such a placement, the thread might be migrated to a different cpuset, or its
cpuset changed to different CPUs or memory nodes, invalidating the CPU or memory
node number it just computed.

This potential race condition is not a significant problem for applications that
only use these \href{\#other-placement-mechanisms}{Other Placement Mechanisms} early in the job run for initial
placement setup, if the job is only migrated by a batch scheduler after it has been
running for a while.

This \textbf{libcpuset} library provides the following mechanisms to support cpuset
relative thread placement that is robust even if the job is being concurrently
migrated such as by a batch scheduler.

If a job needs to pin a thread to a single CPU, then it can use the
convenient \href{\#cpuset-pin}{cpuset{\_}pin} function. This is the most common case.

If a job needs to implement some other variation of placement, such as to
specific memory nodes, or to more than one CPU, then it can use the following
functions to safely guard such code from placement changes caused by job
migration:
\begin{quote}
\begin{itemize}
\item {} 
\href{\#cpuset-get-placement}{cpuset{\_}get{\_}placement}

\item {} 
\href{\#cpuset-equal-placement}{cpuset{\_}equal{\_}placement}

\item {} 
\href{\#cpuset-free-placement}{cpuset{\_}free{\_}placement}

\end{itemize}
\end{quote}

The \textbf{libcpuset} functions \href{\#cpuset-c-rel-to-sys-cpu}{cpuset{\_}c{\_}rel{\_}to{\_}sys{\_}cpu} and variations provide a
convenient means to convert between system wide and cpuset relative CPU and
memory node numbering.


%___________________________________________________________________________

\hypertarget{safe-job-migration}{}
\pdfbookmark[1]{4.4~~~Safe Job Migration}{safe-job-migration}
\subsection*{4.4~~~Safe Job Migration}

Jobs that make proper use of \href{\#cpuset-aware-thread-pinning}{Cpuset Aware Thread Pinning}, rather than the
unsafe \href{\#other-placement-mechanisms}{Other Placement Mechanisms}, can be safely migrated to a different
cpuset, or have their cpuset's CPUs or memory nodes safely changed, without
destroying the per-thread placement done within the job.

A batch scheduler can safely migrate jobs while preserving per-thread placement
of a job that is concurrently using \href{\#cpuset-aware-thread-pinning}{Cpuset Aware Thread Pinning}.

A batch scheduler can accomplish this with the following steps:
\newcounter{listcnt4}
\begin{list}{\arabic{listcnt4}.}
{
\usecounter{listcnt4}
\setlength{\rightmargin}{\leftmargin}
}
\item {} 
Suspend the tasks in the job, perhaps by sending their process group a SIGSTOP.

\item {} 
Use the \href{\#cpuset-init-pidlist}{cpuset{\_}init{\_}pidlist} and related pidlist functions to determine
the list of tasks in the job.

\item {} 
Use \texttt{sched{\_}getaffinity(2)} to query the CPU affinity of each task in the job.

\item {} 
Create a new cpuset, under a temporary name, with the new desired CPU and
memory placement.

\item {} 
Invoke \href{\#cpuset-migrate-all}{cpuset{\_}migrate{\_}all} to move the jobs tasks from the old cpuset to the new.

\item {} 
Use \href{\#cpuset-delete}{cpuset{\_}delete} to delete the old cpuset.

\item {} 
Use \texttt{rename(2)} on the \texttt{/dev/cpuset} based path of the new temporary cpuset to rename
that cpuset to the to the old cpuset name.

\item {} 
Convert the results of the previous \texttt{sched{\_}getaffinity(2)} calls to the new
cpuset placement, preserving cpuset relative offset by using the \href{\#cpuset-c-rel-to-sys-cpu}{cpuset{\_}c{\_}rel{\_}to{\_}sys{\_}cpu}
and related functions.

\item {} 
Use \texttt{sched{\_}setaffinity(2)} to reestablish the per-task CPU binding of each
thread in the job.

\item {} 
Resume the tasks in the job, perhaps by sending their process group a SIGCONT.

\end{list}

The \texttt{sched{\_}getaffinity(2)} and \texttt{sched{\_}setaffinity(2)} 'C' library
calls are limited by 'C' library internals to systems with 1024 CPUs or fewer.
To write code that will work on larger systems, one should use the
\texttt{syscall(2)} indirect system call wrapper to directly invoke the underlying
system call, bypassing the 'C' library API for these calls. Perhaps in the
future the \textbf{libcpuset} library will provide functions that make it easier for
a batch scheduler to obtain, migrate, and set a tasks CPU affinity.

The suspend and resume are required in order to keep tasks in the job from
changing their per thread CPU placement between step 3 and step 6. The kernel
automatically migrates the per-thread memory node placement during step 4, which
it has to as there is no way for one task to modify the NUMA memory placement
policy of another task. But the kernel does not automatically migrate the
per-thread CPU placement, as that can be handled by a user level process such
as a batch scheduler doing the migration, as above.

Migrating a job from a larger cpuset (more CPUs or nodes) to a smaller cpuset
will lose placement information, and subsequently moving that cpuset back to a
larger cpuset will not recover that information. Such migrations lose track of
information on a jobs placement. This loss of information of the jobs CPU
affinity can be avoided as described above, using \texttt{sched{\_}getaffinity(2)} and
\texttt{sched{\_}setaffinity(2)} to save and restore the placement (affinity) across
such a pair of moves. This loss of information of the jobs NUMA memory placement
cannot be avoided because one task (the one doing the migration) cannot save nor
restore the NUMA memory placement policy of another. So if a batch scheduler
wants to migrate jobs without causing them to lose their \texttt{mbind(2)} or
\texttt{set{\_}mempolicy(2)} placement, it should only migrate to cpusets with at least
as many memory nodes as the original cpuset.


%___________________________________________________________________________

\hypertarget{cpus-and-memory-nodes}{}
\pdfbookmark[0]{5~~~CPUs and Memory Nodes}{cpus-and-memory-nodes}
\section*{5~~~CPUs and Memory Nodes}

As of this writing, the Linux kernel NUMA support presumes that there
are some memory nodes and some CPUs, and that for each CPU, there is
exactly one preferred or local memory node. Frequently, multiple (two
or four perhaps) CPUs will be local to the same memory node. Some
memory nodes have no local CPUs - these are called headless nodes.

However, this is not the only possible architecture, and architectures
are constantly changing, usually toward the more complex. Driven by
nonstop increases in logic density for a half century now, the bus,
cache, CPU, memory and storage hierarchy of large systems continues to
evolve.

This cpuset interface should remain stable over a long period of time,
and be usable over a variety of system architectures. So this
interface \emph{avoids} presuming that there is exactly one memory node
local to each CPU. Rather it just presumes that there are some
CPUs and memory nodes, that some zero or more memory nodes are local
to each CPU, and that some zero or more CPUs are local to each memory
node.

This interface provides the functions \href{\#cpuset-localmems}{cpuset{\_}localmems} to identify, for a CPU,
which memory node(s) are local to that CPU, and \href{\#cpuset-localcpus}{cpuset{\_}localcpus} to
identify for a memory node, which CPU(s) are local. Applications using this
interface can explicitly specify just one of CPU or memory placement, and
then use these functions to determine the corresponding memory or CPU placement.

In some situations, applications will want to explicitly place both
CPUs and memory nodes, not necessarily according to the default relation
between a CPU and its local memory node.  Such situations include
the following.
\begin{itemize}
\item {} 
You can control specific CPU and memory node placement when precise
placement control is required, such as when optimizing performance
for a specific important application.

\item {} 
A higher level system service such as a batch scheduler can control
specific CPU and memory node placement.

\item {} 
On systems having some memory on headless nodes (memory nodes with no
associated local CPU), you can explicitly use a particular headless
node with a particular CPU.

\item {} 
On systems supporting Hyper-Threadthreading, one could affectively disable
Hyper-Threading, by using just one out of the two or more execution
engines (CPUs) available on a processor die.

\end{itemize}


%___________________________________________________________________________

\hypertarget{extensible-api}{}
\pdfbookmark[0]{6~~~Extensible API}{extensible-api}
\section*{6~~~Extensible API}

In order to provide for the convenient and robust extensibility of
this cpuset API over time, the following function enables dynamically
obtaining pointers for optional functions by name, at run-time:
\begin{quote}

void *cpuset{\_}function(const char * function{\_}name) - returns function pointer,
or NULL if function name unrecognized
\end{quote}

For maximum portability, you should not reference any optional cpuset function
by name by explicit name.

However, if you are willing to presume that an optional function will
always be available on the target systems of interest, you might decide to
explicitly reference it by name, in order to improve the clarity and simplicity
of the software in question.

Also to support robust extensibility, flags and integer option values have names
dynamically resolved at run-time, not via preprocessor macros.

All functions use only the primitive types of int, char *, pid{\_}t (for process id
of a task), size{\_}t (for buffer sizes), pointers to opaque structures, functions
whose signatures use these types, and pointers to such functions. They use no
structure members, special types or magic define constants.

Some functions in \href{\#advanced-cpuset-library-functions}{Advanced Cpuset Library Functions} are marked \textbf{{[}optional]}. They
are not available in all implementations of libcpuset. Additional \textbf{{[}optional]}
cpuset{\_}* functions may also be added in the future. Functions that
are \emph{not} marked \textbf{{[}optional]} are available on all implementations of
libcpuset.so, and can be called directly without using cpuset{\_}function().
However, any of them \emph{can} also be called indirectly via cpuset{\_}function().

To safely invoke an optional function, such as for example \texttt{cpuset{\_}migrate()},
use the following call sequence:
\begin{quote}{\ttfamily \raggedright \noindent
/*~fp~points~to~function~of~the~type~of~cpuset{\_}migrate()~*/~\\
int~(*fp)(struct~cpuset~*fromcp,~struct~cpuset~*tocp,~pid{\_}t~pid);~\\
fp~=~cpuset{\_}function("cpuset{\_}migrate");~\\
if~(fp)~{\{}~\\
~~~~~~~~fp(~...~);~\\
{\}}~else~{\{}~\\
~~~~~~~~puts~("cpuset~migration~not~supported");~\\
{\}}
}\end{quote}

If you invoke an \textbf{{[}optional]} function directly, then your resulting program
will not be able to link with any version of libcpuset.so that does not define that
particular function.


%___________________________________________________________________________

\hypertarget{cpuset-text-format}{}
\pdfbookmark[0]{7~~~Cpuset Text Format}{cpuset-text-format}
\section*{7~~~Cpuset Text Format}

Cpuset settings may be exported to, and imported from, text files using
a text format representation of cpusets.

The permissions of files holding these text
representations have no special significance to the implementation of cpusets.
Rather, the permissions of the special cpuset files in the cpuset file system,
normally mounted at \texttt{/dev/cpuset}, control reading and writing of and attaching to
cpusets.

The text representation of cpusets is not essential to the use of
cpusets. One can directly manipulate the special files in the cpuset file
system.  This text representation provides an alternative that may be
convenient for some uses.

The cpuset text format supports one directive per line. Comments begin with the
'{\#}' character and extend to the end of line.

After stripping comments, the first white space separated token on each
remaining line selects from the following possible directives:
\begin{quote}
\begin{description}
%[visit_definition_list_item]
\item[cpus] %[visit_definition]

Specify which CPUs are in this cpuset.  The second token
on the line must be a comma-separated list of CPU numbers
and ranges of numbers, optionally modified by a stride operator
(see below).

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[mems] %[visit_definition]

Specify which memory nodes are in this cpuset.  The second
token on the line must be a comma-separated list of memory
node numbers and ranges of numbers, optionally modified by a
stride operator (see below).

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[cpu{\_}exclusive] %[visit_definition]

The cpu{\_}exclusive flag is set.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[mem{\_}exclusive] %[visit_definition]

The mem{\_}exclusive flag is set.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[notify{\_}on{\_}release] %[visit_definition]

The notify{\_}on{\_}release flag is set

%[depart_definition]
%[depart_definition_list_item]
\end{description}
\end{quote}

Additional unnecessary tokens on a line are quietly ignored. Lines containing
only comments and white space are ignored.

The token ``cpu'' is allowed for ``cpus'', and ``mem'' for ``mems''. Matching is case
insensitive.

The \titlereference{stride operator} for ``cpus'' and ``mems'' values is used to designate every
N-th CPU or memory node in a range of numbers.  It is written as a colon
\textbf{``:''} followed by the number N, with no spaces on either side of the colon.
For example, the range \texttt{0-31:2} designates the 16 even numbers 0, 2, 4, ... 30.

The \textbf{libcpuset} routines \href{\#cpuset-import}{cpuset{\_}import} and \href{\#cpuset-export}{cpuset{\_}export} to handle converting
the internal 'struct cpuset' representation of cpusets to (export) and from (import)
this text representation.


%___________________________________________________________________________

\hypertarget{basic-cpuset-library-functions}{}
\pdfbookmark[0]{8~~~Basic Cpuset Library Functions}{basic-cpuset-library-functions}
\section*{8~~~Basic Cpuset Library Functions}

The basic cpuset API provides functions usable from 'C' for processor and
memory placement within a cpuset.

These functions enable an application to place various threads of its execution
on specific CPUs within its current cpuset, and perform related functions such
as asking how large the current cpuset is, and on which CPU within the current
cpuset a thread is currently executing.

These functions do not provide the full power of the advanced cpuset API, but
they are easier to use, and provide some common needs of multi-threaded
applications.

Unlike the rest of this document, the functions described in this section use
cpuset relative numbering. In a cpuset of N CPUs, the relative cpu numbers range
from zero to N - 1.

Unlike the underlying system calls \texttt{sched{\_}setaffinity}, \texttt{mbind} or
\texttt{set{\_}mempolicy}, these basic cpuset API functions are robust in the
presence of cpuset migration.  If you pin a thread in your job to one of
the CPUs in your jobs cpuset, it will stay properly pinned even if your
jobs cpuset is migrated later to another set of CPUs and memory nodes,
or even if the migration is occurring at the same time as your calls.

memory placement is done automatically, preferring the node local to the
requested CPU. Threads may only be placed on a single CPU. This avoids the need
to allocate and free the bitmasks required to specify a set of several CPUs.
These functions do not support creating or removing cpusets, only the placement
of threads within an existing cpuset. This avoids the need to explicitly
allocate and free cpuset structures. Operations only apply to the current
thread, avoiding the need to pass the process id of the thread to be affected.

If more powerful capabilities are needed, use the \href{\#advanced-cpuset-library-functions}{Advanced Cpuset Library
Functions}. These basic functions do not provide any essential new capability.
They are implemented using the advanced functions, and are fully interoperable
with them.

On error, these routines return -1 and set \texttt{errno}. If invoked on an operating
system kernel that does not support cpusets, \texttt{errno} is set to \texttt{ENOSYS}. If invoked
on a system that supports cpusets, but when the cpuset file system is not
currently mounted at \texttt{/dev/cpuset}, then \texttt{errno} is set to \texttt{ENODEV}.

The following inclusion and linkage provides access to the cpuset API
from 'C' code:
\begin{quote}
\begin{quote}{\ttfamily \raggedright \noindent
{\#}include~<cpuset.h>~\\
/*~link~with~-lcpuset~*/
}\end{quote}
\end{quote}

The following functions are supported in the basic cpuset 'C' API:
\begin{quote}
\begin{itemize}
\item {} 
\href{\#cpuset-pin}{cpuset{\_}pin} - Pin the current thread to a CPU, preferring local memory

\item {} 
\href{\#cpuset-size}{cpuset{\_}size} - Return the number of CPUs are in the current tasks cpuset

\item {} 
\href{\#cpuset-where}{cpuset{\_}where} - On which CPU in current tasks cpuset did the task most recently execute

\item {} 
\href{\#cpuset-unpin}{cpuset{\_}unpin} - Remove affect of \href{\#cpuset-pin}{cpuset{\_}pin}, let task have run of its entire cpuset

\end{itemize}
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-pin}{}
\pdfbookmark[1]{8.1~~~cpuset{\_}pin}{cpuset-pin}
\subsection*{8.1~~~cpuset{\_}pin}


%___________________________________________________________________________

\hypertarget{int-cpuset-pin-int-relcpu}{}
\pdfbookmark[2]{int cpuset{\_}pin(int relcpu);}{int-cpuset-pin-int-relcpu}
\subsubsection*{\texttt{int cpuset{\_}pin(int relcpu);}}
\begin{quote}

Pin the current task to execute only on the CPU \texttt{relcpu}, which is a
relative CPU number within the current cpuset of that task. Also,
automatically pin the memory allowed to be used by the current task to
prefer the memory on that same node (as determined by the \href{\#cpuset-cpu2node}{cpuset{\_}cpu2node}
function), but to allow any memory in the cpuset if no free memory is readily
available on the same node.

Return 0 on success, -1 on error. Errors include \texttt{relcpu} being too large
(greater than cpuset{\_}size() - 1). They also include running on a system that
doesn't support cpusets (\texttt{ENOSYS}) and running when the cpuset file system is
not mounted at \texttt{/dev/cpuset} (\texttt{ENODEV}).
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-size}{}
\pdfbookmark[1]{8.2~~~cpuset{\_}size}{cpuset-size}
\subsection*{8.2~~~cpuset{\_}size}


%___________________________________________________________________________

\hypertarget{int-cpuset-size}{}
\pdfbookmark[2]{int cpuset{\_}size();}{int-cpuset-size}
\subsubsection*{\texttt{int cpuset{\_}size();}}
\begin{quote}

Return the number of CPUs in the current tasks cpuset. The relative CPU
numbers that are passed to the \href{\#cpuset-pin}{cpuset{\_}pin} function and that are returned by
the \href{\#cpuset-where}{cpuset{\_}where} function, must be between 0 and N - 1 inclusive, where N
is the value returned by \href{\#cpuset-size}{cpuset{\_}size}.

Returns -1 and sets \texttt{errno} on error. Errors include running on a system that
doesn't support cpusets (\texttt{ENOSYS}) and running when the cpuset file system is
not mounted at \texttt{/dev/cpuset} (\texttt{ENODEV}).
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-where}{}
\pdfbookmark[1]{8.3~~~cpuset{\_}where}{cpuset-where}
\subsection*{8.3~~~cpuset{\_}where}


%___________________________________________________________________________

\hypertarget{int-cpuset-where}{}
\pdfbookmark[2]{int cpuset{\_}where();}{int-cpuset-where}
\subsubsection*{\texttt{int cpuset{\_}where();}}
\begin{quote}

Return the CPU number, relative to the current tasks cpuset, of the CPU on
which the current task most recently executed. If a task is allowed to
execute on more than one CPU, then there is no guarantee that the task is
still executing on the CPU returned by \href{\#cpuset-where}{cpuset{\_}where}, by the time that the
user code obtains the return value.

Returns -1 and sets \texttt{errno} on error. Errors include running on a system that
doesn't support cpusets (\texttt{ENOSYS}) and running when the cpuset file system is
not mounted at \texttt{/dev/cpuset} (\texttt{ENODEV}).
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-unpin}{}
\pdfbookmark[1]{8.4~~~cpuset{\_}unpin}{cpuset-unpin}
\subsection*{8.4~~~cpuset{\_}unpin}


%___________________________________________________________________________

\hypertarget{int-cpuset-unpin}{}
\pdfbookmark[2]{int cpuset{\_}unpin();}{int-cpuset-unpin}
\subsubsection*{\texttt{int cpuset{\_}unpin();}}
\begin{quote}

Remove the CPU and memory pinning affects of any previous \href{\#cpuset-pin}{cpuset{\_}pin} call,
allowing the current task to execute on any CPU in its current cpuset and to
allocate memory on any memory node in its current cpuset. Return -1 on
error, 0 on success.

Returns -1 and sets \texttt{errno} on error. Errors include running on a system that
doesn't support cpusets (\texttt{ENOSYS}) and running when the cpuset file system is
not mounted at \texttt{/dev/cpuset} (\texttt{ENODEV}).
\end{quote}


%___________________________________________________________________________

\hypertarget{using-cpusets-with-hyper-threads}{}
\pdfbookmark[0]{9~~~Using Cpusets with Hyper-Threads}{using-cpusets-with-hyper-threads}
\section*{9~~~Using Cpusets with Hyper-Threads}

\titlereference{Threading} in a software application splits instructions into multiple
streams so that multiple processors can act on them.

Hyper-Threading (HT) Technology, developed by Intel Corporation,
provides thread-level parallelism on each processor, resulting in more
efficient use of processor resources, higher processing throughput,
and improved performance. One physical CPU can appear as two logical CPUs
by having additional registers to overlap two instruction streams or
a single processor can have dual-cores executing instructions in parallel.

In addition to their traditional use to control the placement of
jobs on the CPUs and memory nodes of a system, cpusets also provides
a convenient mechanism to control the use of Hyper-Threading.

Some jobs achieve better performance using both of the Hyper-Thread
sides, A and B, of a processor core, and some run better using just
one of the sides, allowing the other side to idle.

Since each logical (Hyper-Threaded) processor in a core has a distinct
CPU number, one can easily specify a cpuset that contains both sides, or
contains just one side from each of the processor cores in the cpuset.

Cpusets can be configured to include any combination of the logical
CPUs in a system.

For example, the following cpuset configuration file called \texttt{cpuset.cfg}
includes the A sides of an HT enabled system, along with all the memory,
on the first 32 nodes (assuming 2 cores per node). The colon ':' prefixes
the stride.  The stride of '2' in this example means use every other logical
CPU.
\begin{quote}
\begin{quote}{\ttfamily \raggedright \noindent
cpus~0-127:2~~~~{\#}~even~numbered~CPUs~0,~2,~4,~...~126~\\
mems~0-31~~~~~~~{\#}~memory~nodes~0,~1,~2,~...~31
}\end{quote}
\end{quote}

To create a cpuset called \texttt{foo} and run a command called \texttt{bar} in
that cpuset, defined by the cpuset configuration file \texttt{cpuset.cfg}
shown above, use the following commands:
\begin{quote}
\begin{quote}{\ttfamily \raggedright \noindent
cpuset~-c~/foo~<~cpuset.cfg~\\
cpuset~-i~/foo~-I~bar
}\end{quote}
\end{quote}

To specify both sides of the first 64 cores, use the following entry
in your cpuset configuration file:
\begin{quote}
\begin{quote}{\ttfamily \raggedright \noindent
cpus~0-127
}\end{quote}
\end{quote}

To specify just the B sides, use the following entry in your cpuset
configuration file:
\begin{quote}
\begin{quote}{\ttfamily \raggedright \noindent
cpus~1-127:2
}\end{quote}
\end{quote}

The examples above assume that CPUs are uniformly numbered, with the even
numbers for the A side and odd numbers for the B side. This is usually the case,
but not guaranteed. One could still place a job on a system that was not
uniformly numbered, but currently it would involve a longer argument list to the
\texttt{cpus} option, explicitly listing the desired CPUs.

Typically, the logical numbering of CPUs puts the even numbered CPUs on the A
sides, and the odd numbered CPUs on the B side. The stride operator (``:2'', above)
makes it easy to specify that only every other side will be used. If the CPU
number range starts with an even number, this will be the A sides, and if the
range starts with an odd number, this will be the B sides.

A \texttt{ps(1)} or \texttt{top(1)} invocation will show a handful of threads on unused
CPUs, but these are kernel threads assigned to every CPU in support of user
applications running on those CPUs, to handle tasks such as asynchronous file
system writes and task migration between CPUs. If no application is actually
using a CPU, then the kernel threads on that CPU will be almost always idle
and will consume very little memory.


%___________________________________________________________________________

\hypertarget{advanced-cpuset-library-functions}{}
\pdfbookmark[0]{10~~~Advanced Cpuset Library Functions}{advanced-cpuset-library-functions}
\section*{10~~~Advanced Cpuset Library Functions}

The advanced cpuset API provides functions usable from 'C' for managing
cpusets on a system-wide basis.

These functions primarily deal with the three entities (1) \texttt{struct cpuset *},
(2) system cpusets and (3) tasks.
\begin{quote}

The \texttt{struct cpuset *} provides a transient in-memory structure
used to build up a description of an existing or desired cpuset.
These structs can be allocated, freed, queried and modified.

Actual kernel cpusets are created under \texttt{/dev/cpuset}, which is the usual
mount point of the kernels virtual cpuset filesystem. These cpusets are
visible to all tasks (with sufficient authority) in the system, and persist
until the system is rebooted or until the cpuset is explicitly deleted.
These cpusets can be created, deleted, queried, modified, listed and
examined.

Every task (also known as a process) is bound to exactly one
cpuset at a time. You can list which tasks are bound to a given
cpuset, and to which cpuset a given task is bound. You can change
to which cpuset a task is bound.
\end{quote}

The primary attributes of a cpuset are its lists of CPUs and memory
nodes. The scheduling affinity for each task, whether set by default
or explicitly by the \texttt{sched{\_}setaffinity()} system call, is constrained
to those CPUs that are available in that tasks cpuset. The NUMA memory
placement for each task, whether set by default or explicitly by the
\texttt{mbind()} system call, is constrained to those memory nodes that are
available in that tasks cpuset.  This provides the essential purpose of
cpusets - to constrain the CPU and memory usage of tasks to specified
subsets of the system.

The other essential attribute of a cpuset is its pathname beneath
\texttt{/dev/cpuset}. All tasks bound to the same cpuset pathname can be
managed as a unit, and this hierarchical name space describes the
nested resource management and hierarchical permission space supported
by cpusets. Also, this hierarchy is used to enforce strict exclusion,
using the following rules:
\begin{quote}
\begin{itemize}
\item {} 
A cpuset may only be marked strictly exclusive for CPU or memory
if its parent is also.

\item {} 
A cpuset may not make any CPUs or memory nodes available that are
not also available in its parent.

\item {} 
If a cpuset is exclusive for CPU or memory, then it may not overlap
CPUs or memory with any of its siblings.

\end{itemize}
\end{quote}

The combination of these rules enables checking for strict exclusion just by
making various checks on the parent, siblings and existing child cpusets of the
cpuset being changed, without having to check all cpusets in the system.

On error, some of these routines return -1 or NULL and set \texttt{errno}. If one of the
routines below that requires cpuset kernel support is invoked on an operating
system kernel that does not support cpusets, then that routine returns failure
and \texttt{errno} is set to \texttt{ENOSYS}. If invoked on a system that supports
cpusets, but when the cpuset file system is not currently mounted at
\texttt{/dev/cpuset}, then it returns failure and \texttt{errno} is set to \texttt{ENODEV}.

The following inclusion and linkage provides access to the cpuset API
from 'C' code:
\begin{quote}
\begin{quote}{\ttfamily \raggedright \noindent
{\#}include~<bitmask.h>~\\
{\#}include~<cpuset.h>~\\
/*~link~with~-lcpuset~*/
}\end{quote}
\end{quote}

The following functions are supported in the advanced cpuset 'C' API:
\begin{itemize}
\item {} 
Cpuset library (libcpuset) version
\begin{itemize}
\item {} 
\href{\#cpuset-version}{cpuset{\_}version} - \textbf{{[}optional]} Version (simple integer) of the library

\end{itemize}

\item {} 
Allocate and free \texttt{struct cpuset *}
\begin{itemize}
\item {} 
\href{\#cpuset-alloc}{cpuset{\_}alloc} - Return handle to newly allocated \texttt{struct cpuset *}

\item {} 
\href{\#cpuset-free}{cpuset{\_}free} - Discard no longer needed \texttt{struct cpuset *}

\end{itemize}

\item {} 
Lengths of CPUs and memory nodes bitmasks - needed to allocate bitmasks
\begin{itemize}
\item {} 
\href{\#cpuset-cpus-nbits}{cpuset{\_}cpus{\_}nbits} - Number of bits needed for a CPU bitmask on current system

\item {} 
\href{\#cpuset-mems-nbits}{cpuset{\_}mems{\_}nbits} - Number of bits needed for a memory bitmask on current system

\end{itemize}

\item {} 
Set various attributes of a \texttt{struct cpuset *}
\begin{itemize}
\item {} 
\href{\#cpuset-setcpus}{cpuset{\_}setcpus} - Specify CPUs in cpuset

\item {} 
\href{\#cpuset-setmems}{cpuset{\_}setmems} - Specify memory nodes in cpuset

\item {} 
\href{\#cpuset-set-iopt}{cpuset{\_}set{\_}iopt} - Specify an integer value option of cpuset

\item {} 
\href{\#cpuset-set-sopt}{cpuset{\_}set{\_}sopt} - \textbf{{[}optional]} Specify a string value option of cpuset

\end{itemize}

\item {} 
Query various attributes of a \texttt{struct cpuset *}
\begin{itemize}
\item {} 
\href{\#cpuset-getcpus}{cpuset{\_}getcpus} - Query CPUs in cpuset

\item {} 
\href{\#cpuset-getmems}{cpuset{\_}getmems} - Query memory nodes in cpuset

\item {} 
\href{\#cpuset-cpus-weight}{cpuset{\_}cpus{\_}weight} - Number of CPUs in a cpuset

\item {} 
\href{\#cpuset-mems-weight}{cpuset{\_}mems{\_}weight} - Number of memory nodes in a cpuset

\item {} 
\href{\#cpuset-get-iopt}{cpuset{\_}get{\_}iopt} - Query an integer value option of cpuset

\item {} 
\href{\#cpuset-get-sopt}{cpuset{\_}get{\_}sopt} - \textbf{{[}optional]} Query a string value option of cpuset

\end{itemize}

\item {} 
Local CPUs and memory nodes
\begin{itemize}
\item {} 
\href{\#cpuset-localcpus}{cpuset{\_}localcpus} - Query the CPUs local to specified memory nodes

\item {} 
\href{\#cpuset-localmems}{cpuset{\_}localmems} - Query the memory nodes local to specified CPUs

\item {} 
\href{\#cpuset-cpumemdist}{cpuset{\_}cpumemdist} - \textbf{{[}optional]} Hardware distance from CPU to memory node

\item {} 
\href{\#cpuset-cpu2node}{cpuset{\_}cpu2node} - Return system number of memory node closest to specified CPU.

\item {} 
\href{\#cpuset-addr2node}{cpuset{\_}addr2node} - Return sytem number of memory node holding page at specified address.

\end{itemize}

\item {} 
Create, delete, query, modify, list and examine cpusets.
\begin{itemize}
\item {} 
\href{\#cpuset-create}{cpuset{\_}create} - Create named cpuset as specified by struct cpuset *

\item {} 
\href{\#cpuset-delete}{cpuset{\_}delete} - Delete the specified cpuset (if empty)

\item {} 
\href{\#cpuset-query}{cpuset{\_}query} - Set struct cpuset to settings of specified cpuset

\item {} 
\href{\#cpuset-modify}{cpuset{\_}modify} - Modify a cpuset's settings to those specified in a struct cpuset

\item {} 
\href{\#cpuset-getcpusetpath}{cpuset{\_}getcpusetpath} - Get path of a tasks (0 for current) cpuset.

\item {} 
\href{\#cpuset-cpusetofpid}{cpuset{\_}cpusetofpid} - Set struct cpuset to settings of cpuset of specified task

\item {} 
\href{\#cpuset-mountpoint}{cpuset{\_}mountpoint} - Return path at which cpuset filesystem is mounted

\item {} 
\href{\#cpuset-collides-exclusive}{cpuset{\_}collides{\_}exclusive} - \textbf{{[}optional]} True if would collide exclusive

\item {} 
\href{\#cpuset-nuke}{cpuset{\_}nuke} - \textbf{{[}optional]} Remove cpuset anyway possible

\end{itemize}

\item {} 
List tasks currently attached to a cpuset
\begin{itemize}
\item {} 
\href{\#cpuset-init-pidlist}{cpuset{\_}init{\_}pidlist} - Initialize a list of tasks attached to a cpuset

\item {} 
\href{\#cpuset-pidlist-length}{cpuset{\_}pidlist{\_}length} - Return the number of tasks in such a list

\item {} 
\href{\#cpuset-get-pidlist}{cpuset{\_}get{\_}pidlist} - Return a specific task from such a list

\item {} 
\href{\#cpuset-freepidlist}{cpuset{\_}freepidlist} - Deallocate such a list

\end{itemize}

\item {} 
Attach tasks to cpusets.
\begin{itemize}
\item {} 
\href{\#cpuset-move}{cpuset{\_}move} - Move task (0 for current) to a cpuset

\item {} 
\href{\#cpuset-move-all}{cpuset{\_}move{\_}all} - Move all tasks in a list of pids to a cpuset

\item {} 
\href{\#cpuset-move-cpuset-tasks}{cpuset{\_}move{\_}cpuset{\_}tasks} - \textbf{{[}optional]} Move all tasks in a cpuset to another cpuset

\item {} 
\href{\#cpuset-migrate}{cpuset{\_}migrate} - \textbf{{[}optional]} Move a task and its memory to a cpuset

\item {} 
\href{\#cpuset-migrate-all}{cpuset{\_}migrate{\_}all} - \textbf{{[}optional]} Move all tasks with memory in a list of pids to a cpuset

\item {} 
\href{\#cpuset-reattach}{cpuset{\_}reattach} - Rebind \texttt{cpus{\_}allowed} of each task in a cpuset after changing its cpus

\end{itemize}

\item {} 
Determine memory pressure
\begin{itemize}
\item {} 
\href{\#cpuset-open-memory-pressure}{cpuset{\_}open{\_}memory{\_}pressure} - \textbf{{[}optional]} Open handle to read memory{\_}pressure

\item {} 
\href{\#cpuset-read-memory-pressure}{cpuset{\_}read{\_}memory{\_}pressure} - \textbf{{[}optional]} Read cpuset current memory{\_}pressure

\item {} 
\href{\#cpuset-close-memory-pressure}{cpuset{\_}close{\_}memory{\_}pressure} - \textbf{{[}optional]} Close handle to read memory pressure

\end{itemize}

\item {} 
Map between cpuset relative and system-wide CPU and memory node numbers
\begin{itemize}
\item {} 
\href{\#cpuset-c-rel-to-sys-cpu}{cpuset{\_}c{\_}rel{\_}to{\_}sys{\_}cpu} - Map cpuset relative CPU number to system wide number

\item {} 
\href{\#cpuset-c-sys-to-rel-cpu}{cpuset{\_}c{\_}sys{\_}to{\_}rel{\_}cpu} - Map system wide CPU number to cpuset relative number

\item {} 
\href{\#cpuset-c-rel-to-sys-mem}{cpuset{\_}c{\_}rel{\_}to{\_}sys{\_}mem} - Map cpuset relative memory node number to system wide number

\item {} 
\href{\#cpuset-c-sys-to-rel-mem}{cpuset{\_}c{\_}sys{\_}to{\_}rel{\_}mem} - Map system wide memory node number to cpuset relative number

\item {} 
\href{\#cpuset-p-rel-to-sys-cpu}{cpuset{\_}p{\_}rel{\_}to{\_}sys{\_}cpu} - Map task cpuset relative CPU number to system wide number

\item {} 
\href{\#cpuset-p-sys-to-rel-cpu}{cpuset{\_}p{\_}sys{\_}to{\_}rel{\_}cpu} - Map system wide CPU number to task cpuset relative number

\item {} 
\href{\#cpuset-p-rel-to-sys-mem}{cpuset{\_}p{\_}rel{\_}to{\_}sys{\_}mem} - Map task cpuset relative memory node number to system wide number

\item {} 
\href{\#cpuset-p-sys-to-rel-mem}{cpuset{\_}p{\_}sys{\_}to{\_}rel{\_}mem} - Map system wide memory node number to task cpuset relative number

\end{itemize}

\item {} 
Placement operations - for detecting cpuset migration
\begin{itemize}
\item {} 
\href{\#cpuset-get-placement}{cpuset{\_}get{\_}placement} - \textbf{{[}optional]} Return current placement of task pid

\item {} 
\href{\#cpuset-equal-placement}{cpuset{\_}equal{\_}placement} - \textbf{{[}optional]} True if two placements equal

\item {} 
\href{\#cpuset-free-placement}{cpuset{\_}free{\_}placement} - \textbf{{[}optional]} Free placement

\end{itemize}

\item {} 
Traverse a cpuset hierarchy.
\begin{itemize}
\item {} 
\href{\#cpuset-fts-open}{cpuset{\_}fts{\_}open} - \textbf{{[}optional]} Open cpuset hierarchy

\item {} 
\href{\#cpuset-fts-read}{cpuset{\_}fts{\_}read} - \textbf{{[}optional]} Next entry in hierarchy

\item {} 
\href{\#cpuset-fts-reverse}{cpuset{\_}fts{\_}reverse} - \textbf{{[}optional]} Reverse order of cpusets

\item {} 
\href{\#cpuset-fts-rewind}{cpuset{\_}fts{\_}rewind} - \textbf{{[}optional]} Rewind to first cpuset in list

\item {} 
\href{\#cpuset-fts-get-path}{cpuset{\_}fts{\_}get{\_}path} - \textbf{{[}optional]} Get entry's cpuset path

\item {} 
\href{\#cpuset-fts-get-stat}{cpuset{\_}fts{\_}get{\_}stat} - \textbf{{[}optional]} Get entry's stat(2) pointer

\item {} 
\href{\#cpuset-fts-get-cpuset}{cpuset{\_}fts{\_}get{\_}cpuset} - \textbf{{[}optional]} Get entry's cpuset pointer

\item {} 
\href{\#cpuset-fts-get-errno}{cpuset{\_}fts{\_}get{\_}errno} - \textbf{{[}optional]} Get entry's errno

\item {} 
\href{\#cpuset-fts-get-info}{cpuset{\_}fts{\_}get{\_}info} - \textbf{{[}optional]} Get operation causing error

\item {} 
\href{\#cpuset-fts-close}{cpuset{\_}fts{\_}close} - \textbf{{[}optional]} Close cpuset hierarchy

\end{itemize}

\item {} 
Bind to a CPU or memory node within the current cpuset
\begin{itemize}
\item {} 
\href{\#cpuset-cpubind}{cpuset{\_}cpubind} - Bind to a single CPU within a cpuset (uses sched{\_}setaffinity(2))

\item {} 
\href{\#cpuset-latestcpu}{cpuset{\_}latestcpu} - Most recent CPU on which a task has executed

\item {} 
\href{\#cpuset-membind}{cpuset{\_}membind} - Bind to a single memory node within a cpuset (uses set{\_}mempolicy(2))

\end{itemize}

\item {} 
Export cpuset settings to a regular file, and import them from a regular file
\begin{itemize}
\item {} 
\href{\#cpuset-export}{cpuset{\_}export} - Export cpuset settings to a text file

\item {} 
\href{\#cpuset-import}{cpuset{\_}import} - Import cpuset settings from a text file

\end{itemize}

\item {} 
Support calls to \textbf{{[}optional]} cpuset{\_}* API routines
\begin{itemize}
\item {} 
\href{\#cpuset-function}{cpuset{\_}function} - Return pointer to a libcpuset.so function, or \texttt{NULL}

\end{itemize}

\end{itemize}

A typical calling sequence would use the above functions in the following
order to create a new cpuset named ``xyz'' and attach itself to it.
\begin{quote}
\begin{quote}{\ttfamily \raggedright \noindent
struct~cpuset~*cp~=~cpuset{\_}alloc();~\\
various~cpuset{\_}set*(cp,~...)~calls~\\
cpuset{\_}create(cp,~"xyz");~\\
cpuset{\_}free(cp);~~~~~\\
cpuset{\_}move(0,~"xyz");
}\end{quote}
\end{quote}

Some functions above are marked \textbf{{[}optional]}.  For more information, see the \titlereference{Extensible API} section, above, for an explanation of this
marking and how to invoke such functions in a portable manner.


%___________________________________________________________________________

\hypertarget{cpuset-version}{}
\pdfbookmark[1]{10.1~~~cpuset{\_}version}{cpuset-version}
\subsection*{10.1~~~cpuset{\_}version}


%___________________________________________________________________________

\hypertarget{int-cpuset-version}{}
\pdfbookmark[2]{int cpuset{\_}version();}{int-cpuset-version}
\subsubsection*{\texttt{int cpuset{\_}version();}}
\begin{quote}

Version (simple integer) of the cpuset library (\texttt{libcpuset}). The version
number returned by \href{\#cpuset-version}{cpuset{\_}version}() is incremented anytime that any
changes or additions are made to its API or behaviour.  Other mechanims are
provided to maintain full upward compatibility with this libraries API.  This
\href{\#cpuset-version}{cpuset{\_}version}() call is intended to provide a fallback mechanism in case
an application needs to distinguish between two previous versions of this
library.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-alloc}{}
\pdfbookmark[1]{10.2~~~cpuset{\_}alloc}{cpuset-alloc}
\subsection*{10.2~~~cpuset{\_}alloc}


%___________________________________________________________________________

\hypertarget{struct-cpuset-cpuset-alloc}{}
\pdfbookmark[2]{struct cpuset *cpuset{\_}alloc();}{struct-cpuset-cpuset-alloc}
\subsubsection*{\texttt{struct cpuset *cpuset{\_}alloc();}}
\begin{quote}

Creates, initializes and returns a handle to a struct cpuset, which is an
opaque data structure used to describe a cpuset.

After obtaining a struct cpuset handle with this call, one can use the
various \texttt{cpuset{\_}set()} methods to specify which CPUs and memory nodes are
in the cpuset and other attributes. Then one can create such a cpuset with
the \texttt{cpuset{\_}create()} call and free cpuset handles with the \texttt{cpuset{\_}free()}
call.

The \href{\#cpuset-alloc}{cpuset{\_}alloc} function returns a zero pointer (NULL) and sets
\titlereference{errno} in the event that \texttt{malloc(3)} fails. See the \texttt{malloc(3)}
man page for possible values of \titlereference{errno} (\texttt{ENOMEM} being the most
likely).
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-free}{}
\pdfbookmark[1]{10.3~~~cpuset{\_}free}{cpuset-free}
\subsection*{10.3~~~cpuset{\_}free}


%___________________________________________________________________________

\hypertarget{void-cpuset-free-struct-cpuset-cp}{}
\pdfbookmark[2]{void cpuset{\_}free(struct cpuset *cp);}{void-cpuset-free-struct-cpuset-cp}
\subsubsection*{\texttt{void cpuset{\_}free(struct cpuset *cp);}}
\begin{quote}

Frees the memory associated with a struct cpuset handle, which must have
been returned by a previous \texttt{cpuset{\_}alloc()} call.  If cp is NULL, no
operation is performed.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-cpus-nbits}{}
\pdfbookmark[1]{10.4~~~cpuset{\_}cpus{\_}nbits}{cpuset-cpus-nbits}
\subsection*{10.4~~~cpuset{\_}cpus{\_}nbits}


%___________________________________________________________________________

\hypertarget{int-cpuset-cpus-nbits}{}
\pdfbookmark[2]{int cpuset{\_}cpus{\_}nbits();}{int-cpuset-cpus-nbits}
\subsubsection*{\texttt{int cpuset{\_}cpus{\_}nbits();}}
\begin{quote}

Return the number of bits in a CPU bitmask on current system.
Useful when using \texttt{bitmask{\_}alloc()} to allocate a CPU mask.
Some other routines below return \texttt{cpuset{\_}cpus{\_}nbits()} as
an out-of-bounds indicator.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-mems-nbits}{}
\pdfbookmark[1]{10.5~~~cpuset{\_}mems{\_}nbits}{cpuset-mems-nbits}
\subsection*{10.5~~~cpuset{\_}mems{\_}nbits}


%___________________________________________________________________________

\hypertarget{int-cpuset-mems-nbits}{}
\pdfbookmark[2]{int cpuset{\_}mems{\_}nbits();}{int-cpuset-mems-nbits}
\subsubsection*{\texttt{int cpuset{\_}mems{\_}nbits();}}
\begin{quote}

Return the number of bits in a memory node bitmask on current system.
Useful when using \texttt{bitmask{\_}alloc()} to allocate a memory node mask.
Some other routines below return \texttt{cpuset{\_}mems{\_}nbits()} as
an out-of-bounds indicator.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-setcpus}{}
\pdfbookmark[1]{10.6~~~cpuset{\_}setcpus}{cpuset-setcpus}
\subsection*{10.6~~~cpuset{\_}setcpus}


%___________________________________________________________________________

\hypertarget{int-cpuset-setcpus-struct-cpuset-cp-const-struct-bitmask-cpus}{}
\pdfbookmark[2]{int cpuset{\_}setcpus(struct cpuset *cp, const struct bitmask *cpus);}{int-cpuset-setcpus-struct-cpuset-cp-const-struct-bitmask-cpus}
\subsubsection*{\texttt{int cpuset{\_}setcpus(struct cpuset *cp, const struct bitmask *cpus);}}
\begin{quote}

Given a bitmask of CPUs, the cpuset{\_}setcpus() call sets the specified cpuset
\texttt{cp} to include exactly those CPUs.

Returns 0 on success, else -1 on error, setting \texttt{errno}. This routine can
fail if malloc(3) fails.  See the \texttt{malloc(3)} man page for possible values
of \titlereference{errno} (\texttt{ENOMEM} being the most likely).
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-setmems}{}
\pdfbookmark[1]{10.7~~~cpuset{\_}setmems}{cpuset-setmems}
\subsection*{10.7~~~cpuset{\_}setmems}


%___________________________________________________________________________

\hypertarget{void-cpuset-setmems-struct-cpuset-cp-const-struct-bitmask-mems}{}
\pdfbookmark[2]{void cpuset{\_}setmems(struct cpuset *cp, const struct bitmask *mems);}{void-cpuset-setmems-struct-cpuset-cp-const-struct-bitmask-mems}
\subsubsection*{\texttt{void cpuset{\_}setmems(struct cpuset *cp, const struct bitmask *mems);}}
\begin{quote}
\begin{quote}

Given a bitmask of memory nodes, the \texttt{cpuset{\_}setmems()} call sets the
specified cpuset \texttt{cp} to include exactly those memory nodes.
\end{quote}

Returns 0 on success, else -1 on error, setting \texttt{errno}. This routine can
fail if malloc(3) fails.  See the \texttt{malloc(3)} man page for possible values
of \titlereference{errno} (\texttt{ENOMEM} being the most likely).
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-set-iopt}{}
\pdfbookmark[1]{10.8~~~cpuset{\_}set{\_}iopt}{cpuset-set-iopt}
\subsection*{10.8~~~cpuset{\_}set{\_}iopt}


%___________________________________________________________________________

\hypertarget{int-cpuset-set-iopt-struct-cpuset-cp-const-char-optionname-int-value}{}
\pdfbookmark[2]{int cpuset{\_}set{\_}iopt(struct cpuset *cp, const char *optionname, int value);}{int-cpuset-set-iopt-struct-cpuset-cp-const-char-optionname-int-value}
\subsubsection*{\texttt{int cpuset{\_}set{\_}iopt(struct cpuset *cp, const char *optionname, int value);}}
\begin{quote}

Sets cpuset integer valued option \texttt{optionname} to specified integer value.
Returns 0 if \texttt{optionname} is recognized and value is an allowed value for
that option. Returns -1 if \texttt{optionname} is recognized, but value is not
allowed. Returns -2 if \texttt{optionname} is not recognized.  Boolean options
accept any non-zero value as equivalent to a value of one (1).

The following \texttt{optionname}'s are recognized:
\begin{itemize}
\item {} 
\texttt{cpu{\_}exclusive} - sibling cpusets not allowed to overlap cpus
(see section \href{\#exclusive-cpusets}{Exclusive Cpusets}, above)

\item {} 
\texttt{mem{\_}exclusive} - sibling cpusets not allowed to overlap mems
(see section \href{\#exclusive-cpusets}{Exclusive Cpusets}, above)

\item {} 
\texttt{notify{\_}on{\_}release} - invoke \texttt{/sbin/cpuset{\_}release{\_}agent} when
cpuset released (see section \href{\#notify-on-release}{Notify On Release}, above)

\item {} 
\texttt{memory{\_}migrate} - causes memory pages to migrate to new mems
(see section \href{\#memory-migration}{Memory Migration}, above)

\item {} 
\texttt{memory{\_}spread{\_}page} - causes kernel buffer (page) cache to
spread over cpuset (see section \href{\#memory-spread}{Memory Spread}, above)

\item {} 
\texttt{memory{\_}spread{\_}slab} - causes kernel file i/o data (directory and
inode slab caches) to spread over cpuset(see section \href{\#memory-spread}{Memory Spread}, above)

\end{itemize}
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-set-sopt}{}
\pdfbookmark[1]{10.9~~~cpuset{\_}set{\_}sopt}{cpuset-set-sopt}
\subsection*{10.9~~~cpuset{\_}set{\_}sopt}


%___________________________________________________________________________

\hypertarget{int-cpuset-set-sopt-struct-cpuset-cp-const-char-optionname-const-char-value}{}
\pdfbookmark[2]{int cpuset{\_}set{\_}sopt(struct cpuset *cp, const char *optionname, const char *value);}{int-cpuset-set-sopt-struct-cpuset-cp-const-char-optionname-const-char-value}
\subsubsection*{\texttt{int cpuset{\_}set{\_}sopt(struct cpuset *cp, const char *optionname, const char *value);}}
\begin{quote}

Sets cpuset string valued option \texttt{optionname} to specified string value.
Returns 0 if \texttt{optionname} is recognized and value is an allowed value for
that option. Returns -1 if \texttt{optionname} is recognized, but value is not
allowed. Returns -2 if \texttt{optionname} is not recognized.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-getcpus}{}
\pdfbookmark[1]{10.10~~~cpuset{\_}getcpus}{cpuset-getcpus}
\subsection*{10.10~~~cpuset{\_}getcpus}


%___________________________________________________________________________

\hypertarget{int-cpuset-getcpus-const-struct-cpuset-cp-struct-bitmask-cpus}{}
\pdfbookmark[2]{int cpuset{\_}getcpus(const struct cpuset *cp, struct bitmask *cpus);}{int-cpuset-getcpus-const-struct-cpuset-cp-struct-bitmask-cpus}
\subsubsection*{\texttt{int cpuset{\_}getcpus(const struct cpuset *cp, struct bitmask *cpus);}}
\begin{quote}

Query CPUs in cpuset \texttt{cp}, by writing them to the bitmask \texttt{cpus}.
Pass \texttt{cp} == NULL to query the current tasks cpuset

If the CPUs have not been set in cpuset \texttt{cp}, then no operation
is performed, -1 is returned, and \texttt{errno} is set to \texttt{EINVAL}.

Returns 0 on success, else -1 on error, setting \texttt{errno}. This routine can
fail if malloc(3) fails.  See the \texttt{malloc(3)} man page for possible values
of \titlereference{errno} (\texttt{ENOMEM} being the most likely).
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-getmems}{}
\pdfbookmark[1]{10.11~~~cpuset{\_}getmems}{cpuset-getmems}
\subsection*{10.11~~~cpuset{\_}getmems}


%___________________________________________________________________________

\hypertarget{int-cpuset-getmems-const-struct-cpuset-cp-struct-bitmask-mems}{}
\pdfbookmark[2]{int cpuset{\_}getmems(const struct cpuset *cp, struct bitmask *mems);}{int-cpuset-getmems-const-struct-cpuset-cp-struct-bitmask-mems}
\subsubsection*{\texttt{int cpuset{\_}getmems(const struct cpuset *cp, struct bitmask *mems);}}
\begin{quote}

Query memory nodes in cpuset \texttt{cp}, by writing them to the bitmask \texttt{mems}.
Pass \texttt{cp} == NULL to query the current tasks cpuset.

If the memory nodes have not been set in cpuset \texttt{cp}, then no operation
is performed, -1 is returned, and \texttt{errno} is set to \texttt{EINVAL}.

Returns 0 on success, else -1 on error, setting \texttt{errno}. This routine can
fail if malloc(3) fails.  See the \texttt{malloc(3)} man page for possible values
of \titlereference{errno} (\texttt{ENOMEM} being the most likely).
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-cpus-weight}{}
\pdfbookmark[1]{10.12~~~cpuset{\_}cpus{\_}weight}{cpuset-cpus-weight}
\subsection*{10.12~~~cpuset{\_}cpus{\_}weight}


%___________________________________________________________________________

\hypertarget{int-cpuset-cpus-weight-const-struct-cpuset-cp}{}
\pdfbookmark[2]{int cpuset{\_}cpus{\_}weight(const struct cpuset *cp);}{int-cpuset-cpus-weight-const-struct-cpuset-cp}
\subsubsection*{\texttt{int cpuset{\_}cpus{\_}weight(const struct cpuset *cp);}}
\begin{quote}

Query number of CPUs in cpuset \texttt{cp}. Pass \texttt{cp} == NULL to query the
current tasks cpuset

If the CPUs have not been set in cpuset \texttt{cp}, then zero(0) is
returned.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-mems-weight}{}
\pdfbookmark[1]{10.13~~~cpuset{\_}mems{\_}weight}{cpuset-mems-weight}
\subsection*{10.13~~~cpuset{\_}mems{\_}weight}


%___________________________________________________________________________

\hypertarget{int-cpuset-mems-weight-const-struct-cpuset-cp}{}
\pdfbookmark[2]{int cpuset{\_}mems{\_}weight(const struct cpuset *cp);}{int-cpuset-mems-weight-const-struct-cpuset-cp}
\subsubsection*{\texttt{int cpuset{\_}mems{\_}weight(const struct cpuset *cp);}}
\begin{quote}

Query number of memory nodes in cpuset \texttt{cp}.
Pass \texttt{cp} == NULL to query the current tasks cpuset

If the memory nodes have not been set in cpuset \texttt{cp}, then
zero (0) is returned.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-get-iopt}{}
\pdfbookmark[1]{10.14~~~cpuset{\_}get{\_}iopt}{cpuset-get-iopt}
\subsection*{10.14~~~cpuset{\_}get{\_}iopt}


%___________________________________________________________________________

\hypertarget{int-cpuset-get-iopt-const-struct-cpuset-cp-const-char-optionname}{}
\pdfbookmark[2]{int cpuset{\_}get{\_}iopt(const struct cpuset *cp, const char *optionname);}{int-cpuset-get-iopt-const-struct-cpuset-cp-const-char-optionname}
\subsubsection*{\texttt{int cpuset{\_}get{\_}iopt(const struct cpuset *cp, const char *optionname);}}
\begin{quote}

Query value of integer option \texttt{optionname} in cpuset \texttt{cp}. Returns value of
\texttt{optionname} if it is recognized, else returns -1.  Integer values in an uninitialized
cpuset have value 0.

The following \texttt{optionname}'s are recognized:
\begin{itemize}
\item {} 
\texttt{cpu{\_}exclusive} - sibling cpusets not allowed to overlap cpus
(see section \href{\#exclusive-cpusets}{Exclusive Cpusets}, above)

\item {} 
\texttt{mem{\_}exclusive} - sibling cpusets not allowed to overlap mems
(see section \href{\#exclusive-cpusets}{Exclusive Cpusets}, above)

\item {} 
\texttt{notify{\_}on{\_}release} - invoke \texttt{/sbin/cpuset{\_}release{\_}agent} when
cpuset released (see section \href{\#notify-on-release}{Notify On Release}, above)

\item {} 
\texttt{memory{\_}migrate} - causes memory pages to migrate to new mems
(see section \href{\#memory-migration}{Memory Migration}, above)

\item {} 
\texttt{memory{\_}spread{\_}page} - causes kernel buffer (page) cache to
spread over cpuset (see section \href{\#memory-spread}{Memory Spread}, above)

\item {} 
\texttt{memory{\_}spread{\_}slab} - causes kernel file i/o data (directory and
inode slab caches) to spread over cpuset(see section \href{\#memory-spread}{Memory Spread}, above)

\end{itemize}
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-get-sopt}{}
\pdfbookmark[1]{10.15~~~cpuset{\_}get{\_}sopt}{cpuset-get-sopt}
\subsection*{10.15~~~cpuset{\_}get{\_}sopt}


%___________________________________________________________________________

\hypertarget{const-char-cpuset-get-sopt-const-struct-cpuset-cp-const-char-optionname}{}
\pdfbookmark[2]{const char *cpuset{\_}get{\_}sopt(const struct cpuset *cp, const char *optionname);}{const-char-cpuset-get-sopt-const-struct-cpuset-cp-const-char-optionname}
\subsubsection*{\texttt{const char *cpuset{\_}get{\_}sopt(const struct cpuset *cp, const char *optionname);}}
\begin{quote}

Query value of string option \texttt{optionname} in cpuset \texttt{cp}. Returns
pointer to value of \texttt{optionname} if it is recognized, else returns \texttt{NULL}.
String values in an uninitialized cpuset have value \texttt{NULL}.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-localcpus}{}
\pdfbookmark[1]{10.16~~~cpuset{\_}localcpus}{cpuset-localcpus}
\subsection*{10.16~~~cpuset{\_}localcpus}


%___________________________________________________________________________

\hypertarget{int-cpuset-localcpus-const-struct-bitmask-mems-struct-bitmask-cpus}{}
\pdfbookmark[2]{int cpuset{\_}localcpus(const struct bitmask *mems, struct bitmask *cpus);}{int-cpuset-localcpus-const-struct-bitmask-mems-struct-bitmask-cpus}
\subsubsection*{\texttt{int cpuset{\_}localcpus(const struct bitmask *mems, struct bitmask *cpus);}}
\begin{quote}

Query the CPUs local to specified memory nodes \texttt{mems}, by writing them to
the bitmask \texttt{cpus}.  Return 0 on success, -1 on error, setting \texttt{errno}.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-localmems}{}
\pdfbookmark[1]{10.17~~~cpuset{\_}localmems}{cpuset-localmems}
\subsection*{10.17~~~cpuset{\_}localmems}


%___________________________________________________________________________

\hypertarget{int-cpuset-localmems-const-struct-bitmask-cpus-struct-bitmask-mems}{}
\pdfbookmark[2]{int cpuset{\_}localmems(const struct bitmask *cpus, struct bitmask *mems);}{int-cpuset-localmems-const-struct-bitmask-cpus-struct-bitmask-mems}
\subsubsection*{\texttt{int cpuset{\_}localmems(const struct bitmask *cpus, struct bitmask *mems);}}
\begin{quote}

Query the memory nodes local to specified CPUs \texttt{cpus}, by writing them to
the bitmask \texttt{mems}.  Return 0 on success, -1 on error, setting \texttt{errno}.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-cpumemdist}{}
\pdfbookmark[1]{10.18~~~cpuset{\_}cpumemdist}{cpuset-cpumemdist}
\subsection*{10.18~~~cpuset{\_}cpumemdist}


%___________________________________________________________________________

\hypertarget{unsigned-int-cpuset-cpumemdist-int-cpu-int-mem}{}
\pdfbookmark[2]{unsigned int cpuset{\_}cpumemdist(int cpu, int mem);}{unsigned-int-cpuset-cpumemdist-int-cpu-int-mem}
\subsubsection*{\texttt{unsigned int cpuset{\_}cpumemdist(int cpu, int mem);}}
\begin{quote}

Distance between hardware CPU \texttt{cpu} and memory node \texttt{mem}, on a scale
which has the closest distance of a CPU to its local memory valued at ten
(10), and other distances more or less proportional.  Distance is a rough
metric of the bandwidth and delay combined, where a higher distance means
lower bandwidth and longer delays.

If either \texttt{cpu} or \texttt{mem} is not known to the current system, or if any
internal error occurs while evaluating this distance, or if a node has no
CPUs nor memory (I/O only), then the distance returned is UCHAR{\_}MAX (from
limits.h).

These distances are obtained from the systems ACPI SLIT table, and should
conform to:
\begin{quote}

\href{http://h21007.www2.hp.com/dspp/files/unprotected/devresource/Docs/TechPapers/IA64/slit.pdf}{System Locality Information Table Interface Specification} LINEBREAK
Version 1.0, July 25, 2003
\end{quote}

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-cpu2node}{}
\pdfbookmark[1]{10.19~~~cpuset{\_}cpu2node}{cpuset-cpu2node}
\subsection*{10.19~~~cpuset{\_}cpu2node}


%___________________________________________________________________________

\hypertarget{int-cpuset-cpu2node-int-cpu}{}
\pdfbookmark[2]{int cpuset{\_}cpu2node(int cpu);}{int-cpuset-cpu2node-int-cpu}
\subsubsection*{\texttt{int cpuset{\_}cpu2node(int cpu);}}
\begin{quote}

Return system wide number of memory node closest to CPU \texttt{cpu}.
For NUMA architectures common as of this writing, this
would be the number of the node on which \texttt{cpu} is
located.  If an architecture did not have memory on
the same node as a CPU, then it would be the node number
of the memory node closest to or preferred by that \texttt{cpu}.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-addr2node}{}
\pdfbookmark[1]{10.20~~~cpuset{\_}addr2node}{cpuset-addr2node}
\subsection*{10.20~~~cpuset{\_}addr2node}


%___________________________________________________________________________

\hypertarget{int-cpuset-addr2node-void-addr}{}
\pdfbookmark[2]{int cpuset{\_}addr2node(void *addr);}{int-cpuset-addr2node-void-addr}
\subsubsection*{\texttt{int cpuset{\_}addr2node(void *addr);}}
\begin{quote}

Return system wide number of memory node on which is located the
physical page of memory at virtual address \texttt{addr} of
the current tasks address space.  Returns -1 if \texttt{addr}
is not a valid address in the address space of the current
process, with \texttt{errno} set to EFAULT.  If the referenced
physical page was not allocated (faulted in) by the kernel
prior to this call, it will be during the call.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-create}{}
\pdfbookmark[1]{10.21~~~cpuset{\_}create}{cpuset-create}
\subsection*{10.21~~~cpuset{\_}create}


%___________________________________________________________________________

\hypertarget{int-cpuset-create-const-char-cpusetpath-const-struct-cp}{}
\pdfbookmark[2]{int cpuset{\_}create(const char *cpusetpath, const struct *cp);}{int-cpuset-create-const-char-cpusetpath-const-struct-cp}
\subsubsection*{\texttt{int cpuset{\_}create(const char *cpusetpath, const struct *cp);}}
\begin{quote}

Create a cpuset at the specified \texttt{cpusetpath}, as described in the provided
\texttt{struct cpuset *cp}. The parent cpuset of that pathname must already exist.

The parameter \texttt{cp} refers to a handle obtained from a \texttt{cpuset{\_}alloc()}
call.  If the parameter \texttt{cpusetpath} starts with a slash (\texttt{/}) character,
then this a path relative to \texttt{/dev/cpuset}, otherwise it is relative to the
current tasks cpuset.

Returns 0 on success, else -1 on error, setting \texttt{errno}. This routine can
fail if malloc(3) fails.  See the \texttt{malloc(3)} man page for possible values
of \titlereference{errno} (\texttt{ENOMEM} being the most likely).
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-delete}{}
\pdfbookmark[1]{10.22~~~cpuset{\_}delete}{cpuset-delete}
\subsection*{10.22~~~cpuset{\_}delete}


%___________________________________________________________________________

\hypertarget{int-cpuset-delete-const-char-cpusetpath}{}
\pdfbookmark[2]{int cpuset{\_}delete(const char *cpusetpath);}{int-cpuset-delete-const-char-cpusetpath}
\subsubsection*{\texttt{int cpuset{\_}delete(const char *cpusetpath);}}
\begin{quote}

Delete a cpuset at the specified \texttt{cpusetpath}. The cpuset of that pathname must
already exist, be empty (no child cpusets) and be unused (no using tasks).

If the parameter \texttt{cpusetpath} starts with a slash (\texttt{/}) character,
then this a path relative to \texttt{/dev/cpuset}, otherwise it is relative to the
current tasks cpuset.

Returns 0 on success, else -1 on error, setting \texttt{errno}.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-query}{}
\pdfbookmark[1]{10.23~~~cpuset{\_}query}{cpuset-query}
\subsection*{10.23~~~cpuset{\_}query}


%___________________________________________________________________________

\hypertarget{int-cpuset-query-struct-cpuset-cp-const-char-cpusetpath}{}
\pdfbookmark[2]{int cpuset{\_}query(struct cpuset *cp, const char *cpusetpath);}{int-cpuset-query-struct-cpuset-cp-const-char-cpusetpath}
\subsubsection*{\texttt{int cpuset{\_}query(struct cpuset *cp, const char *cpusetpath);}}
\begin{quote}

Set struct cpuset to settings of cpuset at specified path \texttt{cpusetpath}.
\texttt{Struct cpuset *cp} must have been returned by a previous \texttt{cpuset{\_}alloc()}
call. Any previous settings of \texttt{cp} are lost.

If the parameter \texttt{cpusetpath} starts with a slash (\texttt{/}) character,
then this a path relative to \texttt{/dev/cpuset}, otherwise it is relative to the
current tasks cpuset.

Returns 0 on success, or -1 on error, setting \texttt{errno}. Errors include
\texttt{cpusetpath} not referencing a valid cpuset path relative to
\texttt{/dev/cpuset}, or the current task lacking permission to query that cpuset.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-modify}{}
\pdfbookmark[1]{10.24~~~cpuset{\_}modify}{cpuset-modify}
\subsection*{10.24~~~cpuset{\_}modify}


%___________________________________________________________________________

\hypertarget{int-cpuset-modify-const-char-cpusetpath-const-struct-cp}{}
\pdfbookmark[2]{int cpuset{\_}modify(const char *cpusetpath, const struct *cp);}{int-cpuset-modify-const-char-cpusetpath-const-struct-cp}
\subsubsection*{\texttt{int cpuset{\_}modify(const char *cpusetpath, const struct *cp);}}
\begin{quote}

Modify the cpuset at the specified \texttt{cpusetpath}, as described in the provided
\texttt{struct cpuset *cp}. The cpuset at that pathname must already exist.

The parameter \texttt{cp} refers to a handle obtained from a \texttt{cpuset{\_}alloc()} call.

If the parameter \texttt{cpusetpath} starts with a slash (\texttt{/}) character,
then this a path relative to \texttt{/dev/cpuset}, otherwise it is relative to the
current tasks cpuset.

Returns 0 on success, else -1 on error, setting \texttt{errno}.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-getcpusetpath}{}
\pdfbookmark[1]{10.25~~~cpuset{\_}getcpusetpath}{cpuset-getcpusetpath}
\subsection*{10.25~~~cpuset{\_}getcpusetpath}


%___________________________________________________________________________

\hypertarget{char-cpuset-getcpusetpath-pid-t-pid-char-buf-size-t-size}{}
\pdfbookmark[2]{char *cpuset{\_}getcpusetpath(pid{\_}t pid, char *buf, size{\_}t size);}{char-cpuset-getcpusetpath-pid-t-pid-char-buf-size-t-size}
\subsubsection*{\texttt{char *cpuset{\_}getcpusetpath(pid{\_}t pid, char *buf, size{\_}t size);}}
\begin{quote}

The \texttt{cpuset{\_}getcpusetpath()} function copies an absolute pathname of the cpuset to
which task of process id \texttt{pid} is attached, to the array pointed to by
\texttt{buf}, which is of length \texttt{size}.  Use pid == 0 for the current process.

The provided path is relative to the cpuset file system mount point.

If the cpuset path name would require a buffer longer than size elements,
\texttt{NULL} is returned, and \texttt{errno} is set to \texttt{ERANGE} an application should check
for this error, and allocate a larger buffer if necessary.

Returns \texttt{NULL} on failure with \texttt{errno} set accordingly, and buf on success.
The contents of buf are undefined on error.

ERRORS
\begin{quote}

\texttt{EACCES} Permission to read or search a component of the file name was denied.

\texttt{EFAULT} buf points to a bad address.

\texttt{ESRCH}  The pid does not exist.

\texttt{E2BIG}  Larger buffer needed.

\texttt{ENOSYS} Kernel does not support cpusets.
\end{quote}
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-cpusetofpid}{}
\pdfbookmark[1]{10.26~~~cpuset{\_}cpusetofpid}{cpuset-cpusetofpid}
\subsection*{10.26~~~cpuset{\_}cpusetofpid}


%___________________________________________________________________________

\hypertarget{int-cpuset-cpusetofpid-struct-cpuset-cp-pid-t-pid}{}
\pdfbookmark[2]{int cpuset{\_}cpusetofpid(struct cpuset *cp, pid{\_}t pid);}{int-cpuset-cpusetofpid-struct-cpuset-cp-pid-t-pid}
\subsubsection*{\texttt{int cpuset{\_}cpusetofpid(struct cpuset *cp, pid{\_}t pid);}}
\begin{quote}

Set struct cpuset to settings of cpuset to which specified task \texttt{pid} is attached.
\texttt{Struct cpuset *cp} must have been returned by a previous \texttt{cpuset{\_}alloc()}
call. Any previous settings of \texttt{cp} are lost.

Returns 0 on success, or -1 on error, setting \texttt{errno}.

ERRORS
\begin{quote}

\texttt{EACCES} Permission to read or search a component of the file name was denied.

\texttt{EFAULT} buf points to a bad address.

\texttt{ESRCH}  The pid does not exist.

\texttt{ERANGE} Larger buffer needed.

\texttt{ENOSYS} Kernel does not support cpusets.
\end{quote}
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-mountpoint}{}
\pdfbookmark[1]{10.27~~~cpuset{\_}mountpoint}{cpuset-mountpoint}
\subsection*{10.27~~~cpuset{\_}mountpoint}


%___________________________________________________________________________

\hypertarget{const-char-cpuset-mountpoint}{}
\pdfbookmark[2]{const char *cpuset{\_}mountpoint();}{const-char-cpuset-mountpoint}
\subsubsection*{\texttt{const char *cpuset{\_}mountpoint();}}
\begin{quote}

Return the filesystem path at which the cpuset file system is
mounted.  The current implementation of this routine returns
\texttt{/dev/cpuset}, or  the string \texttt{{[}cpuset filesystem not mounted]}
if the cpuset file system is not mounted, or the string
\texttt{{[}cpuset filesystem not supported]} if the system does not
support cpusets.

In general, if the first character of the return string is a
slash (\texttt{/}) then the result is the mount point of the cpuset
file system, otherwise the result is an error message string.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-collides-exclusive}{}
\pdfbookmark[1]{10.28~~~cpuset{\_}collides{\_}exclusive}{cpuset-collides-exclusive}
\subsection*{10.28~~~cpuset{\_}collides{\_}exclusive}


%___________________________________________________________________________

\hypertarget{int-cpuset-collides-exclusive-const-char-cpusetpath-const-struct-cp}{}
\pdfbookmark[2]{int cpuset{\_}collides{\_}exclusive(const char *cpusetpath, const struct *cp);}{int-cpuset-collides-exclusive-const-char-cpusetpath-const-struct-cp}
\subsubsection*{\texttt{int cpuset{\_}collides{\_}exclusive(const char *cpusetpath, const struct *cp);}}
\begin{quote}

Return true (1) if cpuset \texttt{cp} would collide with any sibling of the
cpuset at \texttt{cpusetpath} due to overlap of \texttt{cpu{\_}exclusive} \texttt{cpus} or
\texttt{mem{\_}exclusive} \texttt{mems}.  Return false (0) if no collision, or for
any error.

\href{\#cpuset-create}{cpuset{\_}create} fails with \texttt{errno} == \texttt{EINVAL} if the requested
cpuset would overlap with any sibling, where either one is \texttt{cpu{\_}exclusive}
or \texttt{mem{\_}exclusive}. This is a common, and not obvious error.
\texttt{cpuset{\_}collides{\_}exclusive()} checks for this particular
case, so that code creating cpusets can better identify the situation,
perhaps to issue a more informative error message.

Can also be used to diagnose \href{\#cpuset-modify}{cpuset{\_}modify} failures.  This
routine ignores any existing cpuset with the same path as the
given \texttt{cpusetpath}, and only looks for exclusive collisions with
sibling cpusets of that path.

In case of any error, returns (0) -{}- does not collide.  Presumably
any actual attempt to create or modify a cpuset will encounter the
same error, and report it usefully.

This routine is not particularly efficient; most likely code creating or
modifying a cpuset will want to try the operation first, and then if that
fails with \texttt{errno EINVAL}, perhaps call this routine to determine if an
exclusive cpuset collision caused the error.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-nuke}{}
\pdfbookmark[1]{10.29~~~cpuset{\_}nuke}{cpuset-nuke}
\subsection*{10.29~~~cpuset{\_}nuke}


%___________________________________________________________________________

\hypertarget{int-cpuset-nuke-const-char-cpusetpath-unsigned-int-seconds}{}
\pdfbookmark[2]{int cpuset{\_}nuke(const char *cpusetpath, unsigned int seconds);}{int-cpuset-nuke-const-char-cpusetpath-unsigned-int-seconds}
\subsubsection*{\texttt{int cpuset{\_}nuke(const char *cpusetpath, unsigned int seconds);}}
\begin{quote}

Remove a cpuset, including killing tasks in it, and
removing any descendent cpusets and killing their tasks.

Tasks can take a long time (minutes on some configurations)
to exit.  Loop up to \texttt{seconds} seconds, trying to kill them.

The following steps are taken to remove a cpuset:
\begin{quote}
\newcounter{listcnt5}
\begin{list}{\arabic{listcnt5}.}
{
\usecounter{listcnt5}
\setlength{\rightmargin}{\leftmargin}
}
\item {} 
First, kill all the pids, looping until there are
no more pids in this cpuset or below, or until the
'seconds' timeout limit is exceeded.

\item {} 
Then depth first recursively rmdir the cpuset directories.

\item {} 
If by this point the original cpuset is gone, return success.

\end{list}
\end{quote}

If the timeout is exceeded, and tasks still exist, fail with
errno == ETIME.

This routine sleeps a variable amount of time.  After the first attempt to
kill all the tasks in the cpuset or its descendents, it sleeps one
second, the next time two seconds, increasing one second each loop
up to a max of ten seconds.  If more loops past ten are required
to kill all the tasks, it sleeps ten seconds each subsequent loop.
In any case, before the last loop, it sleeps however many seconds
remain of the original timeout \texttt{seconds} requested.  The total
time of all sleeps will be no more than the requested \texttt{seconds}.

If the cpuset started out empty of any tasks, or if the passed in
\texttt{seconds} was zero, then this routine will return quickly, having
not slept at all.  Otherwise, this routine will at a minimum send
a \texttt{SIGKILL} to all the tasks in this cpuset subtree, then sleep one
second, before looking to see if any tasks remain.  If tasks remain
in the cpuset subtree, and a longer \texttt{seconds} timeout was requested
(more than one), it will continue to kill remaining tasks and sleep,
in a loop, for as long as time and tasks remain.

The signal sent for the kill is hardcoded to \texttt{SIGKILL}.  If some
other signal should be sent first, use a separate code loop,
perhaps based on cpuset{\_}init{\_}pidlist and cpuset{\_}get{\_}pidlist, to
scan the task pids in a cpuset.  If \texttt{SIGKILL} should -not- be sent,
this \texttt{cpuset{\_}nuke()} routine can still be called to recursively
remove a cpuset subtree, by specifying a timeout of zero \texttt{seconds}.

On success, returns 0 with \texttt{errno} == 0.

On failure, returns -1, setting \texttt{errno}.

ERRORS
\begin{quote}

\texttt{EACCES}  search permission denied on intervening directory

\texttt{ETIME}   timed out - tasks remain after 'seconds' timeout

\texttt{EMFILE}  too many open files

\texttt{ENODEV}  /dev/cpuset not mounted

\texttt{ENOENT}  component of cpuset path doesn't exist

\texttt{ENOMEM}  out of memory

\texttt{ENOSYS}  kernel doesn't support cpusets

\texttt{ENOTDIR} component of cpuset path is not a directory

\texttt{EPERM}   lacked permission to kill a task

\texttt{EPERM}   lacked permission to read cpusets or files therein
\end{quote}

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-init-pidlist}{}
\pdfbookmark[1]{10.30~~~cpuset{\_}init{\_}pidlist}{cpuset-init-pidlist}
\subsection*{10.30~~~cpuset{\_}init{\_}pidlist}


%___________________________________________________________________________

\hypertarget{struct-cpuset-pidlist-cpuset-init-pidlist-const-char-cpusetpath-int-recursiveflag}{}
\pdfbookmark[2]{struct cpuset{\_}pidlist *cpuset{\_}init{\_}pidlist(const char *cpusetpath, int recursiveflag);}{struct-cpuset-pidlist-cpuset-init-pidlist-const-char-cpusetpath-int-recursiveflag}
\subsubsection*{\texttt{struct cpuset{\_}pidlist *cpuset{\_}init{\_}pidlist(const char *cpusetpath, int recursiveflag);}}
\begin{quote}

Initialize and return a list of tasks (PIDs) attached to cpuset \texttt{cpusetpath}.
If \texttt{recursiveflag} is zero, include only the tasks directly in that cpuset,
otherwise include all tasks in that cpuset or any descendant thereof.

Beware that tasks may come and go from a cpuset, after this call is made.

If the parameter \texttt{cpusetpath} starts with a slash (\texttt{/}) character,
then this a path relative to \texttt{/dev/cpuset}, otherwise it is relative to the
current tasks cpuset.

On error, return \texttt{NULL} and set \texttt{errno}.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-pidlist-length}{}
\pdfbookmark[1]{10.31~~~cpuset{\_}pidlist{\_}length}{cpuset-pidlist-length}
\subsection*{10.31~~~cpuset{\_}pidlist{\_}length}


%___________________________________________________________________________

\hypertarget{int-cpuset-pidlist-length-const-struct-cpuset-pidlist-pl}{}
\pdfbookmark[2]{int cpuset{\_}pidlist{\_}length(const struct cpuset{\_}pidlist *pl);}{int-cpuset-pidlist-length-const-struct-cpuset-pidlist-pl}
\subsubsection*{\texttt{int cpuset{\_}pidlist{\_}length(const struct cpuset{\_}pidlist *pl);}}
\begin{quote}

Return the number of elements (PIDs) in cpuset{\_}pidlist \texttt{pl}.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-get-pidlist}{}
\pdfbookmark[1]{10.32~~~cpuset{\_}get{\_}pidlist}{cpuset-get-pidlist}
\subsection*{10.32~~~cpuset{\_}get{\_}pidlist}


%___________________________________________________________________________

\hypertarget{pid-t-cpuset-get-pidlist-const-struct-cpuset-pidlist-pl-int-i}{}
\pdfbookmark[2]{pid{\_}t cpuset{\_}get{\_}pidlist(const struct cpuset{\_}pidlist *pl, int i);}{pid-t-cpuset-get-pidlist-const-struct-cpuset-pidlist-pl-int-i}
\subsubsection*{\texttt{pid{\_}t cpuset{\_}get{\_}pidlist(const struct cpuset{\_}pidlist *pl, int i);}}
\begin{quote}

Return the i'th element of a \texttt{cpuset{\_}pidlist}. The elements of a
\texttt{cpuset{\_}pidlist} of length \texttt{N} are numbered \texttt{0} through \texttt{N-1}.
Return \texttt{(pid{\_}t)-1} for any other index \texttt{i}.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-freepidlist}{}
\pdfbookmark[1]{10.33~~~cpuset{\_}freepidlist}{cpuset-freepidlist}
\subsection*{10.33~~~cpuset{\_}freepidlist}


%___________________________________________________________________________

\hypertarget{void-cpuset-freepidlist-struct-cpuset-pidlist-pl}{}
\pdfbookmark[2]{void cpuset{\_}freepidlist(struct cpuset{\_}pidlist *pl);}{void-cpuset-freepidlist-struct-cpuset-pidlist-pl}
\subsubsection*{\texttt{void cpuset{\_}freepidlist(struct cpuset{\_}pidlist *pl);}}
\begin{quote}

Deallocate a list of attached pids.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-move}{}
\pdfbookmark[1]{10.34~~~cpuset{\_}move}{cpuset-move}
\subsection*{10.34~~~cpuset{\_}move}


%___________________________________________________________________________

\hypertarget{int-cpuset-move-pid-t-p-const-char-cpusetpath}{}
\pdfbookmark[2]{int cpuset{\_}move(pid{\_}t p, const char *cpusetpath);}{int-cpuset-move-pid-t-p-const-char-cpusetpath}
\subsubsection*{\texttt{int cpuset{\_}move(pid{\_}t p, const char *cpusetpath);}}
\begin{quote}

Move task whose process id is \texttt{p} to cpuset \texttt{cpusetpath}.
If \texttt{pid} is zero, then the current task is moved.

If the parameter \texttt{cpusetpath} starts with a slash (\texttt{/}) character,
then this a path relative to \texttt{/dev/cpuset}, otherwise it is relative to the
current tasks cpuset.

Returns 0 on success, else -1 on error, setting \texttt{errno}.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-move-all}{}
\pdfbookmark[1]{10.35~~~cpuset{\_}move{\_}all}{cpuset-move-all}
\subsection*{10.35~~~cpuset{\_}move{\_}all}


%___________________________________________________________________________

\hypertarget{int-cpuset-move-all-struct-cpuset-pid-list-pl-const-char-cpusetpath}{}
\pdfbookmark[2]{int cpuset{\_}move{\_}all(struct cpuset{\_}pid{\_}list *pl, const char *cpusetpath);}{int-cpuset-move-all-struct-cpuset-pid-list-pl-const-char-cpusetpath}
\subsubsection*{\texttt{int cpuset{\_}move{\_}all(struct cpuset{\_}pid{\_}list *pl, const char *cpusetpath);}}
\begin{quote}

Move all tasks in list \texttt{pl} to cpuset \texttt{cpusetpath}.

If the parameter \texttt{cpusetpath} starts with a slash (\texttt{/}) character,
then this a path relative to \texttt{/dev/cpuset}, otherwise it is relative to the
current tasks cpuset.

Returns 0 on success, else -1 on error, setting \texttt{errno}.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-move-cpuset-tasks}{}
\pdfbookmark[1]{10.36~~~cpuset{\_}move{\_}cpuset{\_}tasks}{cpuset-move-cpuset-tasks}
\subsection*{10.36~~~cpuset{\_}move{\_}cpuset{\_}tasks}


%___________________________________________________________________________

\hypertarget{int-cpuset-move-cpuset-tasks-const-char-fromrelpath-const-char-torelpath}{}
\pdfbookmark[2]{int cpuset{\_}move{\_}cpuset{\_}tasks(const char *fromrelpath, const char *torelpath);}{int-cpuset-move-cpuset-tasks-const-char-fromrelpath-const-char-torelpath}
\subsubsection*{\texttt{int cpuset{\_}move{\_}cpuset{\_}tasks(const char *fromrelpath, const char *torelpath);}}
\begin{quote}

Move all tasks in cpuset \texttt{fromrelpath} to cpuset \texttt{torelpath}. This may
race with tasks being added to or forking into \texttt{fromrelpath}. Loop
repeatedly, reading the tasks file of cpuset \texttt{fromrelpath} and writing any
task pid's found there to the tasks file of cpuset \texttt{torelpath}, up to ten
attempts, or until the \texttt{tasks} file of cpuset \texttt{fromrelpath} is empty, or
until the cpuset \texttt{fromrelpath} is no longer present.

Returns 0 with errno == 0 if able to empty the tasks file of cpuset
\texttt{fromrelpath}. Of course it is still possible that some independent task
could add another task to cpuset \texttt{fromrelpath} at the same time that such
a successful result is being returned, so there can be no guarantee that a
successful return means that \texttt{fromrelpath} is still empty of tasks.

The cpuset \texttt{fromrelpath} might disappear during this operation, perhaps
because it has \texttt{notify{\_}on{\_}release} set and was automatically removed as
soon as its last task was detached from it. Consider a missing
\texttt{fromrelpath} to be a successful move.

If called with \texttt{fromrelpath} and \texttt{torelpath} pathnames that evaluate to
the same cpuset, then treat that as if \href{\#cpuset-reattach}{cpuset{\_}reattach}() was called,
rebinding each task in this cpuset one time, and return success or failure
depending on the return of that \href{\#cpuset-reattach}{cpuset{\_}reattach}() call.

On failure, returns -1, setting \texttt{errno}.

ERRORS
\begin{quote}

\texttt{EACCES}     search permission denied on intervening directory

\texttt{ENOTEMPTY}  tasks remain after multiple attempts to move them

\texttt{EMFILE}     too many open files

\texttt{ENODEV}     /dev/cpuset not mounted

\texttt{ENOENT}     component of cpuset path doesn't exist

\texttt{ENOMEM}     out of memory

\texttt{ENOSYS}     kernel doesn't support cpusets

\texttt{ENOTDIR}    component of cpuset path is not a directory

\texttt{EPERM}      lacked permission to read cpusets or files therein

\texttt{EACCES`      lacked permission to write a cpuset `{}`tasks} file
\end{quote}

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-migrate}{}
\pdfbookmark[1]{10.37~~~cpuset{\_}migrate}{cpuset-migrate}
\subsection*{10.37~~~cpuset{\_}migrate}


%___________________________________________________________________________

\hypertarget{int-cpuset-migrate-pid-t-pid-const-char-cpusetpath}{}
\pdfbookmark[2]{int cpuset{\_}migrate(pid{\_}t pid, const char *cpusetpath);}{int-cpuset-migrate-pid-t-pid-const-char-cpusetpath}
\subsubsection*{\texttt{int cpuset{\_}migrate(pid{\_}t pid, const char *cpusetpath);}}
\begin{quote}

Migrate task whose process id is \texttt{p} to cpuset \texttt{cpusetpath},
moving its currently allocated memory to nodes in that cpuset,
if not already there.  If \texttt{pid} is zero, then the current task
is migrated.

If the parameter \texttt{cpusetpath} starts with a slash (\texttt{/}) character,
then this a path relative to \texttt{/dev/cpuset}, otherwise it is relative to the
current tasks cpuset.

Returns 0 on success, else -1 on error, setting \texttt{errno}

For more information, see the \titlereference{Memory Migration} section, above.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-migrate-all}{}
\pdfbookmark[1]{10.38~~~cpuset{\_}migrate{\_}all}{cpuset-migrate-all}
\subsection*{10.38~~~cpuset{\_}migrate{\_}all}


%___________________________________________________________________________

\hypertarget{int-cpuset-migrate-all-struct-cpuset-pid-list-pl-const-char-cpusetpath}{}
\pdfbookmark[2]{int cpuset{\_}migrate{\_}all(struct cpuset{\_}pid{\_}list *pl, const char *cpusetpath);}{int-cpuset-migrate-all-struct-cpuset-pid-list-pl-const-char-cpusetpath}
\subsubsection*{\texttt{int cpuset{\_}migrate{\_}all(struct cpuset{\_}pid{\_}list *pl, const char *cpusetpath);}}
\begin{quote}

Move all tasks in list \texttt{pl} to cpuset \texttt{cpusetpath}, moving their
currently allocated memory to nodes in that cpuset, if not already there.

If the parameter \texttt{cpusetpath} starts with a slash (\texttt{/}) character,
then this a path relative to \texttt{/dev/cpuset}, otherwise it is relative to the
current tasks cpuset.

For more information, see the \titlereference{Memory Migration} section, above.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-reattach}{}
\pdfbookmark[1]{10.39~~~cpuset{\_}reattach}{cpuset-reattach}
\subsection*{10.39~~~cpuset{\_}reattach}


%___________________________________________________________________________

\hypertarget{int-cpuset-reattach-const-char-cpusetpath}{}
\pdfbookmark[2]{int cpuset{\_}reattach(const char *cpusetpath);}{int-cpuset-reattach-const-char-cpusetpath}
\subsubsection*{\texttt{int cpuset{\_}reattach(const char *cpusetpath);}}
\begin{quote}

Reattach all tasks in cpuset \texttt{cpusetpath} to itself. This additional step
is necessary anytime that the \texttt{cpus} of a cpuset have been changed, in
order to rebind the \texttt{cpus{\_}allowed} of each task in the cpuset to the new
value. This routine writes the pid of each task currently attached to the
named cpuset to the \texttt{tasks} file of that cpuset. If additional tasks are
being spawned too rapidly into the cpuset at the same time, there is an
unavoidable race condition, and some tasks may be missed.

If the parameter \texttt{cpusetpath} starts with a slash (\texttt{/}) character,
then this a path relative to \texttt{/dev/cpuset}, otherwise it is relative to the
current tasks cpuset.

Returns 0 on success, else -1 on error, setting \texttt{errno}.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-open-memory-pressure}{}
\pdfbookmark[1]{10.40~~~cpuset{\_}open{\_}memory{\_}pressure}{cpuset-open-memory-pressure}
\subsection*{10.40~~~cpuset{\_}open{\_}memory{\_}pressure}


%___________________________________________________________________________

\hypertarget{int-cpuset-open-memory-pressure-const-char-cpusetpath}{}
\pdfbookmark[2]{int cpuset{\_}open{\_}memory{\_}pressure(const char *cpusetpath);}{int-cpuset-open-memory-pressure-const-char-cpusetpath}
\subsubsection*{\texttt{int cpuset{\_}open{\_}memory{\_}pressure(const char *cpusetpath);}}
\begin{quote}

Open a file descriptor from which to read the \texttt{memory{\_}pressure} of
the cpuset \texttt{cpusetpath}.

If the parameter \texttt{cpusetpath} starts with a slash (\texttt{/}) character,
then this a path relative to \texttt{/dev/cpuset}, otherwise it is relative to the
current tasks cpuset.

By default, computation by the kernel of \texttt{memory{\_}pressure} is disabled.
Set the \texttt{memory{\_}pressure{\_}enabled} flag in the top cpuset to enable it.

On error, return \texttt{-1} and set \texttt{errno}.

For more information, see the \titlereference{Memory Pressure} section, above.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-read-memory-pressure}{}
\pdfbookmark[1]{10.41~~~cpuset{\_}read{\_}memory{\_}pressure}{cpuset-read-memory-pressure}
\subsection*{10.41~~~cpuset{\_}read{\_}memory{\_}pressure}


%___________________________________________________________________________

\hypertarget{int-cpuset-read-memory-pressure-int-fd}{}
\pdfbookmark[2]{int cpuset{\_}read{\_}memory{\_}pressure(int fd);}{int-cpuset-read-memory-pressure-int-fd}
\subsubsection*{\texttt{int cpuset{\_}read{\_}memory{\_}pressure(int fd);}}
\begin{quote}

Read and return the current memory{\_}pressure of the cpuset
for which file descriptor \texttt{fd} was opened using \texttt{cpuset{\_}open{\_}memory{\_}pressure}.

Uses the system call \texttt{pread(2)}.  On success, returns a non-negative number,
as described in section \href{\#memory-pressure}{Memory Pressure}.  On failure, returns \texttt{-1} and sets
\texttt{errno}.

By default, computation by the kernel of \texttt{memory{\_}pressure} is disabled.
Set the \texttt{memory{\_}pressure{\_}enabled} flag in the top cpuset to enable it.

For more information, see the \titlereference{Memory Pressure} section, above.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-close-memory-pressure}{}
\pdfbookmark[1]{10.42~~~cpuset{\_}close{\_}memory{\_}pressure}{cpuset-close-memory-pressure}
\subsection*{10.42~~~cpuset{\_}close{\_}memory{\_}pressure}


%___________________________________________________________________________

\hypertarget{void-cpuset-close-memory-pressure-int-fd}{}
\pdfbookmark[2]{void cpuset{\_}close{\_}memory{\_}pressure(int fd);}{void-cpuset-close-memory-pressure-int-fd}
\subsubsection*{\texttt{void cpuset{\_}close{\_}memory{\_}pressure(int fd);}}
\begin{quote}

Close the file descriptor \texttt{fd} which was opened using \texttt{cpuset{\_}open{\_}memory{\_}pressure}.

If \texttt{fd} is not a valid open file descriptor, then this call does nothing.
No error is returned in any case.

By default, computation by the kernel of \texttt{memory{\_}pressure} is disabled.
Set the \texttt{memory{\_}pressure{\_}enabled} flag in the top cpuset to enable it.

For more information, see the \titlereference{Memory Pressure} section, above.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-c-rel-to-sys-cpu}{}
\pdfbookmark[1]{10.43~~~cpuset{\_}c{\_}rel{\_}to{\_}sys{\_}cpu}{cpuset-c-rel-to-sys-cpu}
\subsection*{10.43~~~cpuset{\_}c{\_}rel{\_}to{\_}sys{\_}cpu}


%___________________________________________________________________________

\hypertarget{int-cpuset-c-rel-to-sys-cpu-const-struct-cpuset-cp-int-cpu}{}
\pdfbookmark[2]{int cpuset{\_}c{\_}rel{\_}to{\_}sys{\_}cpu(const struct cpuset *cp, int cpu);}{int-cpuset-c-rel-to-sys-cpu-const-struct-cpuset-cp-int-cpu}
\subsubsection*{\texttt{int cpuset{\_}c{\_}rel{\_}to{\_}sys{\_}cpu(const struct cpuset *cp, int cpu);}}
\begin{quote}

Return the system-wide CPU number that is used by the \texttt{cpu}-th CPU
of the specified cpuset \texttt{cp}.  Returns result of \texttt{cpuset{\_}cpus{\_}nbits()}
if \texttt{cpu} is not in the range {[}\texttt{0}, \texttt{bitmask{\_}weight(cpuset{\_}getcpus(cp))}).
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-c-sys-to-rel-cpu}{}
\pdfbookmark[1]{10.44~~~cpuset{\_}c{\_}sys{\_}to{\_}rel{\_}cpu}{cpuset-c-sys-to-rel-cpu}
\subsection*{10.44~~~cpuset{\_}c{\_}sys{\_}to{\_}rel{\_}cpu}


%___________________________________________________________________________

\hypertarget{int-cpuset-c-sys-to-rel-cpu-const-struct-cpuset-cp-int-cpu}{}
\pdfbookmark[2]{int cpuset{\_}c{\_}sys{\_}to{\_}rel{\_}cpu(const struct cpuset *cp, int cpu);}{int-cpuset-c-sys-to-rel-cpu-const-struct-cpuset-cp-int-cpu}
\subsubsection*{\texttt{int cpuset{\_}c{\_}sys{\_}to{\_}rel{\_}cpu(const struct cpuset *cp, int cpu);}}
\begin{quote}

Return the \texttt{cpu}-th CPU of the specified cpuset \texttt{cp} that is
used by the system-wide CPU number.  Returns result of \texttt{cpuset{\_}cpus{\_}nbits()}
if \texttt{bitmask{\_}isbitset(cpuset{\_}getcpus(cp), cpu)} is false.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-c-rel-to-sys-mem}{}
\pdfbookmark[1]{10.45~~~cpuset{\_}c{\_}rel{\_}to{\_}sys{\_}mem}{cpuset-c-rel-to-sys-mem}
\subsection*{10.45~~~cpuset{\_}c{\_}rel{\_}to{\_}sys{\_}mem}


%___________________________________________________________________________

\hypertarget{int-cpuset-c-rel-to-sys-mem-const-struct-cpuset-cp-int-mem}{}
\pdfbookmark[2]{int cpuset{\_}c{\_}rel{\_}to{\_}sys{\_}mem(const struct cpuset *cp, int mem);}{int-cpuset-c-rel-to-sys-mem-const-struct-cpuset-cp-int-mem}
\subsubsection*{\texttt{int cpuset{\_}c{\_}rel{\_}to{\_}sys{\_}mem(const struct cpuset *cp, int mem);}}
\begin{quote}

Return the system-wide memory node number that is used by the \texttt{mem}-th
memory node of the specified cpuset \texttt{cp}.  Returns result of \texttt{cpuset{\_}mems{\_}nbits()}
if \texttt{mem} is not in the range {[}\texttt{0}, \texttt{bitmask{\_}weight(cpuset{\_}getmems(cp))}).
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-c-sys-to-rel-mem}{}
\pdfbookmark[1]{10.46~~~cpuset{\_}c{\_}sys{\_}to{\_}rel{\_}mem}{cpuset-c-sys-to-rel-mem}
\subsection*{10.46~~~cpuset{\_}c{\_}sys{\_}to{\_}rel{\_}mem}


%___________________________________________________________________________

\hypertarget{int-cpuset-c-sys-to-rel-mem-const-struct-cpuset-cp-int-mem}{}
\pdfbookmark[2]{int cpuset{\_}c{\_}sys{\_}to{\_}rel{\_}mem(const struct cpuset *cp, int mem);}{int-cpuset-c-sys-to-rel-mem-const-struct-cpuset-cp-int-mem}
\subsubsection*{\texttt{int cpuset{\_}c{\_}sys{\_}to{\_}rel{\_}mem(const struct cpuset *cp, int mem);}}
\begin{quote}

Return the \texttt{mem}-th memory node of the specified cpuset \texttt{cp} that is
used by the system-wide memory node number.  Returns result of \texttt{cpuset{\_}mems{\_}nbits()}
if \texttt{bitmask{\_}isbitset(cpuset{\_}getmems(cp), mem)} is false.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-p-rel-to-sys-cpu}{}
\pdfbookmark[1]{10.47~~~cpuset{\_}p{\_}rel{\_}to{\_}sys{\_}cpu}{cpuset-p-rel-to-sys-cpu}
\subsection*{10.47~~~cpuset{\_}p{\_}rel{\_}to{\_}sys{\_}cpu}


%___________________________________________________________________________

\hypertarget{int-cpuset-p-rel-to-sys-cpu-pid-t-pid-int-cpu}{}
\pdfbookmark[2]{int cpuset{\_}p{\_}rel{\_}to{\_}sys{\_}cpu(pid{\_}t pid, int cpu);}{int-cpuset-p-rel-to-sys-cpu-pid-t-pid-int-cpu}
\subsubsection*{\texttt{int cpuset{\_}p{\_}rel{\_}to{\_}sys{\_}cpu(pid{\_}t pid, int cpu);}}
\begin{quote}

Return the system-wide CPU number that is used by the \texttt{cpu}-th CPU
of the cpuset to which task \texttt{pid} is attached.  Returns result of \texttt{cpuset{\_}cpus{\_}nbits()}
if that cpuset doesn't encompass that relative cpu number.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-p-sys-to-rel-cpu}{}
\pdfbookmark[1]{10.48~~~cpuset{\_}p{\_}sys{\_}to{\_}rel{\_}cpu}{cpuset-p-sys-to-rel-cpu}
\subsection*{10.48~~~cpuset{\_}p{\_}sys{\_}to{\_}rel{\_}cpu}


%___________________________________________________________________________

\hypertarget{int-cpuset-p-sys-to-rel-cpu-pid-t-pid-int-cpu}{}
\pdfbookmark[2]{int cpuset{\_}p{\_}sys{\_}to{\_}rel{\_}cpu(pid{\_}t pid, int cpu);}{int-cpuset-p-sys-to-rel-cpu-pid-t-pid-int-cpu}
\subsubsection*{\texttt{int cpuset{\_}p{\_}sys{\_}to{\_}rel{\_}cpu(pid{\_}t pid, int cpu);}}
\begin{quote}

Return the \texttt{cpu}-th CPU of the cpuset to which task \texttt{pid} is attached that is
used by the system-wide CPU number.  Returns result of \texttt{cpuset{\_}cpus{\_}nbits()}
if that cpuset doesn't encompass that system-wide cpu number.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-p-rel-to-sys-mem}{}
\pdfbookmark[1]{10.49~~~cpuset{\_}p{\_}rel{\_}to{\_}sys{\_}mem}{cpuset-p-rel-to-sys-mem}
\subsection*{10.49~~~cpuset{\_}p{\_}rel{\_}to{\_}sys{\_}mem}


%___________________________________________________________________________

\hypertarget{int-cpuset-p-rel-to-sys-mem-pid-t-pid-int-mem}{}
\pdfbookmark[2]{int cpuset{\_}p{\_}rel{\_}to{\_}sys{\_}mem(pid{\_}t pid, int mem);}{int-cpuset-p-rel-to-sys-mem-pid-t-pid-int-mem}
\subsubsection*{\texttt{int cpuset{\_}p{\_}rel{\_}to{\_}sys{\_}mem(pid{\_}t pid, int mem);}}
\begin{quote}

Return the system-wide memory node number that is used by the \texttt{mem}-th
memory node of the cpuset to which task \texttt{pid} is attached.  Returns result
of \texttt{cpuset{\_}mems{\_}nbits()} if that cpuset doesn't encompass that relative
memory node number.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-p-sys-to-rel-mem}{}
\pdfbookmark[1]{10.50~~~cpuset{\_}p{\_}sys{\_}to{\_}rel{\_}mem}{cpuset-p-sys-to-rel-mem}
\subsection*{10.50~~~cpuset{\_}p{\_}sys{\_}to{\_}rel{\_}mem}


%___________________________________________________________________________

\hypertarget{int-cpuset-p-sys-to-rel-mem-pid-t-pid-int-mem}{}
\pdfbookmark[2]{int cpuset{\_}p{\_}sys{\_}to{\_}rel{\_}mem(pid{\_}t pid, int mem);}{int-cpuset-p-sys-to-rel-mem-pid-t-pid-int-mem}
\subsubsection*{\texttt{int cpuset{\_}p{\_}sys{\_}to{\_}rel{\_}mem(pid{\_}t pid, int mem);}}
\begin{quote}

Return the \texttt{mem}-th memory node of the cpuset to which task \texttt{pid} is attached that is
used by the system-wide memory node number.  Returns result of \texttt{cpuset{\_}mems{\_}nbits()}
if that cpuset doesn't encompass that system-wide memory node.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-get-placement}{}
\pdfbookmark[1]{10.51~~~cpuset{\_}get{\_}placement}{cpuset-get-placement}
\subsection*{10.51~~~cpuset{\_}get{\_}placement}

cpuset{\_}get{\_}placement(pid) - {[}optional Return current placement of task pid
\begin{quote}

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.

This function returns an opaque \texttt{struct placement *} pointer. The results
of calling \href{\#cpuset-get-placement}{cpuset{\_}get{\_}placement}() twice at different points in a program
can be compared by calling \href{\#cpuset-equal-placement}{cpuset{\_}equal{\_}placement}() to determine if the
specified task has had its cpuset CPU and memory placement modified between
those two \href{\#cpuset-get-placement}{cpuset{\_}get{\_}placement}() calls.

When finished with a \texttt{struct placement *} pointer, free it by calling
\href{\#cpuset-free-placement}{cpuset{\_}free{\_}placement}().
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-equal-placement}{}
\pdfbookmark[1]{10.52~~~cpuset{\_}equal{\_}placement}{cpuset-equal-placement}
\subsection*{10.52~~~cpuset{\_}equal{\_}placement}

cpuset{\_}equal{\_}placement(plc1, plc2) - {[}optional] True if two placements equal
\begin{quote}

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.

This function compares two \texttt{struct placement *} pointers, returned by two
separate calls to \href{\#cpuset-get-placement}{cpuset{\_}get{\_}placement}(). This is done to determine if the
specified task has had its cpuset CPU and memory placement modified between
those two \href{\#cpuset-get-placement}{cpuset{\_}get{\_}placement}() calls.

When finished with a \texttt{struct placement *} pointer, free it by calling
\href{\#cpuset-free-placement}{cpuset{\_}free{\_}placement}().

Two \texttt{struct placement *} pointers will compare equal if they have the same
CPU placement \texttt{cpus}, the same memory placement \texttt{mems}, and the same
cpuset path.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-free-placement}{}
\pdfbookmark[1]{10.53~~~cpuset{\_}free{\_}placement}{cpuset-free-placement}
\subsection*{10.53~~~cpuset{\_}free{\_}placement}

cpuset{\_}free{\_}placement(plc) - {[}optional] Free placement
\begin{quote}

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.

Use this routine to free a \texttt{struct placement *} pointer returned by a
previous call to \href{\#cpuset-get-placement}{cpuset{\_}get{\_}placement}().
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-fts-open}{}
\pdfbookmark[1]{10.54~~~cpuset{\_}fts{\_}open}{cpuset-fts-open}
\subsection*{10.54~~~cpuset{\_}fts{\_}open}


%___________________________________________________________________________

\hypertarget{struct-cpuset-fts-tree-cpuset-fts-open-const-char-cpusetpath}{}
\pdfbookmark[2]{struct cpuset{\_}fts{\_}tree *cpuset{\_}fts{\_}open(const char *cpusetpath);}{struct-cpuset-fts-tree-cpuset-fts-open-const-char-cpusetpath}
\subsubsection*{\texttt{struct cpuset{\_}fts{\_}tree *cpuset{\_}fts{\_}open(const char *cpusetpath);}}
\begin{quote}

Open a cpuset hierarchy. Returns a pointer to a \texttt{cpuset{\_}fts{\_}tree}
structure, which can be used to traverse all cpusets below the specified
cpuset \texttt{cpusetpath}.

If the parameter \texttt{cpusetpath} starts with a slash (\texttt{/}) character, then
this path is relative to \texttt{/dev/cpuset}, otherwise it is relative to the
current tasks cpuset.

The \href{\#cpuset-fts-open}{cpuset{\_}fts{\_}open} routine is implemented internally using the
\texttt{fts(3)} library routines for traversing a file hierarchy. The entire
cpuset subtree below \texttt{cpusetpath} is traversed as part of the
\href{\#cpuset-fts-open}{cpuset{\_}fts{\_}open}() call, and all cpuset state and directory stat
information is captured at that time. The other \texttt{cpuset{\_}fts{\_}*} routines
just access this captured state. Any changes to the traversed cpusets made
after the return of the \href{\#cpuset-fts-open}{cpuset{\_}fts{\_}open}() call will not be visible via
the returned \texttt{cpuset{\_}fts{\_}tree} structure.

Internally, the \texttt{fts(3)} options \texttt{FTS{\_}NOCHDIR} and \texttt{FTS{\_}XDEV} are
used, to avoid changing the invoking tasks current directory, and to avoid
descending into any other file systems mounted below \texttt{/dev/cpuset}. The
order in which cpusets will be returned by the \href{\#cpuset-fts-read}{cpuset{\_}fts{\_}read} routine
corresponds to the fts \texttt{pre-order} (FTS{\_}D) visitation order. The internal
fts scan by \href{\#cpuset-fts-open}{cpuset{\_}fts{\_}open} ignores the \texttt{post-order} (FTS{\_}DP)
results.

Because the \href{\#cpuset-fts-open}{cpuset{\_}fts{\_}open}() call collects all the information at once
from an entire cpuset subtree, a simple error return would not provide
sufficient information as to what failed, and on what cpuset in the subtree.
So, except for \texttt{malloc(3)} failures, errors are captured in the list of
entries.

See \href{\#cpuset-fts-get-info}{cpuset{\_}fts{\_}get{\_}info} for details of the \texttt{info} field.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-fts-read}{}
\pdfbookmark[1]{10.55~~~cpuset{\_}fts{\_}read}{cpuset-fts-read}
\subsection*{10.55~~~cpuset{\_}fts{\_}read}


%___________________________________________________________________________

\hypertarget{const-struct-cpuset-fts-entry-cpuset-fts-read-struct-cpuset-fts-tree-cs-tree}{}
\pdfbookmark[2]{const struct cpuset{\_}fts{\_}entry *cpuset{\_}fts{\_}read(struct cpuset{\_}fts{\_}tree *cs{\_}tree);}{const-struct-cpuset-fts-entry-cpuset-fts-read-struct-cpuset-fts-tree-cs-tree}
\subsubsection*{\texttt{const struct cpuset{\_}fts{\_}entry *cpuset{\_}fts{\_}read(struct cpuset{\_}fts{\_}tree *cs{\_}tree);}}
\begin{quote}

Returns next \texttt{cs{\_}entry} in \texttt{cpuset{\_}fts{\_}tree} \texttt{cs{\_}tree} obtained from an
\href{\#cpuset-fts-open}{cpuset{\_}fts{\_}open}() call. One \texttt{cs{\_}entry} is returned for each cpuset
directory that was found in the subtree scanned by the \href{\#cpuset-fts-open}{cpuset{\_}fts{\_}open}()
call.  Use the \texttt{info} field obtained from a \href{\#cpuset-fts-get-info}{cpuset{\_}fts{\_}get{\_}info}()
call to determine which fields of a particular \texttt{cs{\_}entry} are valid,
and which fields contain error information or are not valid.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-fts-reverse}{}
\pdfbookmark[1]{10.56~~~cpuset{\_}fts{\_}reverse}{cpuset-fts-reverse}
\subsection*{10.56~~~cpuset{\_}fts{\_}reverse}


%___________________________________________________________________________

\hypertarget{void-cpuset-fts-reverse-struct-cpuset-fts-tree-cs-tree}{}
\pdfbookmark[2]{void cpuset{\_}fts{\_}reverse(struct cpuset{\_}fts{\_}tree *cs{\_}tree);}{void-cpuset-fts-reverse-struct-cpuset-fts-tree-cs-tree}
\subsubsection*{\texttt{void cpuset{\_}fts{\_}reverse(struct cpuset{\_}fts{\_}tree *cs{\_}tree);}}
\begin{quote}

Reverse order of \texttt{cs{\_}entry}'s in the \texttt{cpuset{\_}fts{\_}tree} \texttt{cs{\_}tree}
obtained from a \href{\#cpuset-fts-open}{cpuset{\_}fts{\_}open}() call.

An open \texttt{cpuset{\_}fts{\_}tree} stores a list of \texttt{cs{\_}entry} cpuset entries, in
pre-order, meaning that a series of \href{\#cpuset-fts-read}{cpuset{\_}fts{\_}read}() calls will always
return a parent cpuset before any of its child cpusets.  Following a
\href{\#cpuset-fts-reverse}{cpuset{\_}fts{\_}reverse}() call, the order of cpuset entries is reversed, putting
it in post-order, so that a series of \href{\#cpuset-fts-read}{cpuset{\_}fts{\_}read}() calls will always
return any children cpusets before their parent cpuset.  A second
\href{\#cpuset-fts-reverse}{cpuset{\_}fts{\_}reverse}() call would put the list back in pre-order again.

To avoid exposing confusing inner details of the implementation across the
API, a \href{\#cpuset-fts-rewind}{cpuset{\_}fts{\_}rewind}() call is always automatically performed on a
\texttt{cpuset{\_}fts{\_}tree} whenever \href{\#cpuset-fts-reverse}{cpuset{\_}fts{\_}reverse}() is called on it.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-fts-rewind}{}
\pdfbookmark[1]{10.57~~~cpuset{\_}fts{\_}rewind}{cpuset-fts-rewind}
\subsection*{10.57~~~cpuset{\_}fts{\_}rewind}


%___________________________________________________________________________

\hypertarget{void-cpuset-fts-rewind-struct-cpuset-fts-tree-cs-tree}{}
\pdfbookmark[2]{void cpuset{\_}fts{\_}rewind(struct cpuset{\_}fts{\_}tree *cs{\_}tree);}{void-cpuset-fts-rewind-struct-cpuset-fts-tree-cs-tree}
\subsubsection*{\texttt{void cpuset{\_}fts{\_}rewind(struct cpuset{\_}fts{\_}tree *cs{\_}tree);}}
\begin{quote}

Rewind a cpuset tree \texttt{cs{\_}tree} obtained from a \href{\#cpuset-fts-open}{cpuset{\_}fts{\_}open}()
call, so that subsequent \href{\#cpuset-fts-read}{cpuset{\_}fts{\_}read}() calls start from the
beginning again.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-fts-get-path}{}
\pdfbookmark[1]{10.58~~~cpuset{\_}fts{\_}get{\_}path}{cpuset-fts-get-path}
\subsection*{10.58~~~cpuset{\_}fts{\_}get{\_}path}


%___________________________________________________________________________

\hypertarget{const-char-cpuset-fts-get-path-const-struct-cpuset-fts-entry-cs-entry}{}
\pdfbookmark[2]{const char *cpuset{\_}fts{\_}get{\_}path(const struct cpuset{\_}fts{\_}entry *cs{\_}entry);}{const-char-cpuset-fts-get-path-const-struct-cpuset-fts-entry-cs-entry}
\subsubsection*{\texttt{const char *cpuset{\_}fts{\_}get{\_}path(const struct cpuset{\_}fts{\_}entry *cs{\_}entry);}}
\begin{quote}

Return the cpuset path, relative to \texttt{/dev/cpuset}, as nul-terminated
string, of a \texttt{cs{\_}entry} obtained from a \href{\#cpuset-fts-read}{cpuset{\_}fts{\_}read}() call.

The results of this call are valid for all \texttt{cs{\_}entry}'s returned from
\href{\#cpuset-fts-read}{cpuset{\_}fts{\_}read}() calls, regardless of the value returned by \href{\#cpuset-fts-get-info}{cpuset{\_}fts{\_}get{\_}info}()
for that \texttt{cs{\_}entry}.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-fts-get-stat}{}
\pdfbookmark[1]{10.59~~~cpuset{\_}fts{\_}get{\_}stat}{cpuset-fts-get-stat}
\subsection*{10.59~~~cpuset{\_}fts{\_}get{\_}stat}


%___________________________________________________________________________

\hypertarget{const-struct-stat-cpuset-fts-get-stat-const-struct-cpuset-fts-entry-cs-entry}{}
\pdfbookmark[2]{const struct stat *cpuset{\_}fts{\_}get{\_}stat(const struct cpuset{\_}fts{\_}entry *cs{\_}entry);}{const-struct-stat-cpuset-fts-get-stat-const-struct-cpuset-fts-entry-cs-entry}
\subsubsection*{\texttt{const struct stat *cpuset{\_}fts{\_}get{\_}stat(const struct cpuset{\_}fts{\_}entry *cs{\_}entry);}}
\begin{quote}

Return pointer to \texttt{stat(2)} information about the cpuset directory of a
\texttt{cs{\_}entry} obtained from a \href{\#cpuset-fts-read}{cpuset{\_}fts{\_}read}() call.

The results of this call are valid for all \texttt{cs{\_}entry}'s returned from
\href{\#cpuset-fts-read}{cpuset{\_}fts{\_}read}() calls, regardless of the value returned by \href{\#cpuset-fts-get-info}{cpuset{\_}fts{\_}get{\_}info}()
for that \texttt{cs{\_}entry}, \textbf{except} in the cases that:
\begin{quote}
\begin{itemize}
\item {} 
the \texttt{info} field returned by \href{\#cpuset-fts-get-info}{cpuset{\_}fts{\_}get{\_}info} contains
\texttt{CPUSET{\_}FTS{\_}ERR{\_}DNR}, in which case, a directory in the path to the
cpuset could not be read and this call will return a \texttt{NULL} pointer, or

\item {} 
the \texttt{info} field returned by \href{\#cpuset-fts-get-info}{cpuset{\_}fts{\_}get{\_}info} contains
\texttt{CPUSET{\_}FTS{\_}ERR{\_}STAT}, in which case a \texttt{stat(2)} failed on this cpuset
directory and this call will return a pointer to a \texttt{struct stat}
containing all zeros.

\end{itemize}
\end{quote}

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-fts-get-cpuset}{}
\pdfbookmark[1]{10.60~~~cpuset{\_}fts{\_}get{\_}cpuset}{cpuset-fts-get-cpuset}
\subsection*{10.60~~~cpuset{\_}fts{\_}get{\_}cpuset}


%___________________________________________________________________________

\hypertarget{const-struct-cpuset-cpuset-fts-get-cpuset-const-struct-cpuset-fts-entry-cs-entry}{}
\pdfbookmark[2]{const struct cpuset *cpuset{\_}fts{\_}get{\_}cpuset(const struct cpuset{\_}fts{\_}entry *cs{\_}entry);}{const-struct-cpuset-cpuset-fts-get-cpuset-const-struct-cpuset-fts-entry-cs-entry}
\subsubsection*{\texttt{const struct cpuset *cpuset{\_}fts{\_}get{\_}cpuset(const struct cpuset{\_}fts{\_}entry *cs{\_}entry);}}
\begin{quote}

Return the \texttt{struct cpuset} pointer of a \texttt{cs{\_}entry} obtained
from a \href{\#cpuset-fts-read}{cpuset{\_}fts{\_}read}() call.  The \texttt{struct cpuset} so referenced
describes the cpuset represented by one directory in the cpuset
hierarchy, and can be used with various other calls in this library.

The results of this call are only valid for a \texttt{cs{\_}entry} if the
\href{\#cpuset-fts-get-info}{cpuset{\_}fts{\_}get{\_}info}() call returns \texttt{CPUSET{\_}FTS{\_}CPUSET} for the \texttt{info} field
of a \texttt{cs{\_}entry}.   If the \texttt{info} field contained \texttt{CPUSET{\_}FTS{\_}ERR{\_}CPUSET},
then \href{\#cpuset-fts-get-cpuset}{cpuset{\_}fts{\_}get{\_}cpuset} returns a pointer to a \texttt{struct cpuset} that
is all zeros.  If the \texttt{info} field contains any other \texttt{CPUSET{\_}FTS{\_}ERR{\_}*} value,
then \href{\#cpuset-fts-get-cpuset}{cpuset{\_}fts{\_}get{\_}cpuset} returns a NULL pointer.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-fts-get-errno}{}
\pdfbookmark[1]{10.61~~~cpuset{\_}fts{\_}get{\_}errno}{cpuset-fts-get-errno}
\subsection*{10.61~~~cpuset{\_}fts{\_}get{\_}errno}


%___________________________________________________________________________

\hypertarget{int-cpuset-fts-get-errno-const-struct-cpuset-fts-entry-cs-entry}{}
\pdfbookmark[2]{int cpuset{\_}fts{\_}get{\_}errno(const struct cpuset{\_}fts{\_}entry *cs{\_}entry);}{int-cpuset-fts-get-errno-const-struct-cpuset-fts-entry-cs-entry}
\subsubsection*{\texttt{int cpuset{\_}fts{\_}get{\_}errno(const struct cpuset{\_}fts{\_}entry *cs{\_}entry);}}
\begin{quote}

Return the \texttt{err} field of a \texttt{cs{\_}entry} obtained from a
\href{\#cpuset-fts-read}{cpuset{\_}fts{\_}read}() call.

If an entry (obtained from \href{\#cpuset-fts-read}{cpuset{\_}fts{\_}read}) has one of the
\texttt{CPUSET{\_}FTS{\_}ERR{\_}*} values in the \texttt{info} field (as described in
\href{\#cpuset-fts-get-info}{cpuset{\_}fts{\_}get{\_}info}), then this \texttt{err} field captures the failing
\texttt{errno} value for that operation. If an entry has the value
\texttt{CPUSET{\_}FTS{\_}CPUSET} in its \texttt{info} field, then this \texttt{err} field will
have the value \texttt{0}.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-fts-get-info}{}
\pdfbookmark[1]{10.62~~~cpuset{\_}fts{\_}get{\_}info}{cpuset-fts-get-info}
\subsection*{10.62~~~cpuset{\_}fts{\_}get{\_}info}


%___________________________________________________________________________

\hypertarget{int-cpuset-fts-get-info-const-struct-cpuset-fts-entry-cs-entry}{}
\pdfbookmark[2]{int cpuset{\_}fts{\_}get{\_}info(const struct cpuset{\_}fts{\_}entry *cs{\_}entry);}{int-cpuset-fts-get-info-const-struct-cpuset-fts-entry-cs-entry}
\subsubsection*{\texttt{int cpuset{\_}fts{\_}get{\_}info(const struct cpuset{\_}fts{\_}entry *cs{\_}entry);}}
\begin{quote}

Return the \texttt{info} field of a \texttt{cs{\_}entry} obtained from a
\href{\#cpuset-fts-read}{cpuset{\_}fts{\_}read}() call.

If this \texttt{info} field has one of the following \texttt{CPUSET{\_}FTS{\_}ERR{\_}*} values,
then it indicates which operation failed, the \texttt{err} field (returned by
\href{\#cpuset-fts-get-errno}{cpuset{\_}fts{\_}get{\_}errno}) captures the failing \texttt{errno} value for that
operation, the \texttt{path} field (returned by \href{\#cpuset-fts-get-path}{cpuset{\_}fts{\_}get{\_}path}) indicates
which cpuset failed, and some of the other entry fields may not be valid,
depending on the value. If an entry has the value \texttt{CPUSET{\_}FTS{\_}CPUSET} for its
\texttt{info} field, then the \texttt{err} field will have the value \texttt{0}, and the
other fields will be contain valid information about that cpuset.

\texttt{info} field values:
\begin{quote}
\begin{description}
%[visit_definition_list_item]
\item[\texttt{CPUSET{\_}FTS{\_}CPUSET = 0}:] %[visit_definition]

Valid cpuset

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[\texttt{CPUSET{\_}FTS{\_}ERR{\_}DNR = 1}:] %[visit_definition]

Error - couldn't read directory

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[\texttt{CPUSET{\_}FTS{\_}ERR{\_}STAT = 2}:] %[visit_definition]

Error - couldn't stat directory

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[\texttt{CPUSET{\_}FTS{\_}ERR{\_}CPUSET = 3}:] %[visit_definition]

Error - \href{\#cpuset-query}{cpuset{\_}query} failed

%[depart_definition]
%[depart_definition_list_item]
\end{description}
\end{quote}

The above \texttt{info} field values are defined using an anonymous \emph{enum} in the
\texttt{cpuset.h} header file. If it necessary to maintain source code
compatibility with earlier versions of the \texttt{cpuset.h} header file lacking
the above \texttt{CPUSET{\_}FTS{\_}*} values, one can conditionally check that the \texttt{C}
preprocessor symbol \texttt{CPUSET{\_}FTS{\_}INFO{\_}VALUES{\_}DEFINED} is not defined
and provide alternative coding for that case.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-fts-close}{}
\pdfbookmark[1]{10.63~~~cpuset{\_}fts{\_}close}{cpuset-fts-close}
\subsection*{10.63~~~cpuset{\_}fts{\_}close}


%___________________________________________________________________________

\hypertarget{void-cpuset-fts-close-struct-cpuset-fts-tree-cs-tree}{}
\pdfbookmark[2]{void cpuset{\_}fts{\_}close(struct cpuset{\_}fts{\_}tree *cs{\_}tree);}{void-cpuset-fts-close-struct-cpuset-fts-tree-cs-tree}
\subsubsection*{\texttt{void cpuset{\_}fts{\_}close(struct cpuset{\_}fts{\_}tree *cs{\_}tree);}}
\begin{quote}

Close a \texttt{cs{\_}tree} obtained from a \href{\#cpuset-fts-open}{cpuset{\_}fts{\_}open}() call, freeing
any internally allocated memory for that \texttt{cs{\_}tree}.

This is an \textbf{{[}optional]} function.   Use \href{\#cpuset-function}{cpuset{\_}function} to invoke it.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-cpubind}{}
\pdfbookmark[1]{10.64~~~cpuset{\_}cpubind}{cpuset-cpubind}
\subsection*{10.64~~~cpuset{\_}cpubind}


%___________________________________________________________________________

\hypertarget{int-cpuset-cpubind-int-cpu}{}
\pdfbookmark[2]{int cpuset{\_}cpubind(int cpu);}{int-cpuset-cpubind-int-cpu}
\subsubsection*{\texttt{int cpuset{\_}cpubind(int cpu);}}
\begin{quote}

Bind the scheduling of the current task to CPU \texttt{cpu}, using the
sched{\_}setaffinity(2) system call.

Fails with a return of -1, and \texttt{errno} set to \texttt{EINVAL}, if \texttt{cpu} is not allowed in
the current tasks cpuset.

The following code will bind the scheduling of a thread to the n-th CPU of the
current cpuset:
\begin{quote}
\begin{quote}{\ttfamily \raggedright \noindent
/*~\\
~*~Confine~current~task~to~only~run~on~the~n-th~CPU~\\
~*~of~its~current~cpuset.~In~a~cpuset~of~N~CPUs,~\\
~*~valid~values~for~n~are~0~..~N-1.~\\
~*/~\\
cpuset{\_}cpubind(cpuset{\_}p{\_}rel{\_}to{\_}sys{\_}cpu(0,~n));
}\end{quote}
\end{quote}
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-latestcpu}{}
\pdfbookmark[1]{10.65~~~cpuset{\_}latestcpu}{cpuset-latestcpu}
\subsection*{10.65~~~cpuset{\_}latestcpu}


%___________________________________________________________________________

\hypertarget{int-cpuset-latestcpu-pid-t-pid}{}
\pdfbookmark[2]{int cpuset{\_}latestcpu(pid{\_}t pid);}{int-cpuset-latestcpu-pid-t-pid}
\subsubsection*{\texttt{int cpuset{\_}latestcpu(pid{\_}t pid);}}
\begin{quote}

Return the most recent CPU on which the specified task \texttt{pid} executed.
If pid is 0, examine current task.

The \texttt{cpuset{\_}latestcpu()} call returns the number of the CPU on which the
specified task \texttt{pid} most recently executed. If a process can be scheduled
on two or more CPUs, then the results of \texttt{cpuset{\_}lastcpu()} may become
invalid even before the query returns to the invoking user code.

The last used CPU is visible for a given pid as field {\#}39 (starting with {\#}1)
in the file /proc/pid/stat. Currently this file has 41 fields, so its the
3rd to the last field.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-membind}{}
\pdfbookmark[1]{10.66~~~cpuset{\_}membind}{cpuset-membind}
\subsection*{10.66~~~cpuset{\_}membind}


%___________________________________________________________________________

\hypertarget{int-cpuset-membind-int-mem}{}
\pdfbookmark[2]{int cpuset{\_}membind(int mem);}{int-cpuset-membind-int-mem}
\subsubsection*{\texttt{int cpuset{\_}membind(int mem);}}
\begin{quote}

Bind the memory allocation of the current task to memory node \texttt{mem}, using
the set{\_}mempolicy(2) system call with a policy of MPOL{\_}BIND.

Fails with a return of -1, and \texttt{errno} set to \texttt{EINVAL}, if \texttt{mem} is not allowed in
the current tasks cpuset.

The following code will bind the memory allocation of a thread to the n-th
memory node of the current cpuset:
\begin{quote}
\begin{quote}{\ttfamily \raggedright \noindent
/*~\\
~*~Confine~current~task~to~only~allocate~memory~on~\\
~*~n-th~node~of~its~current~cpuset.~~In~a~cpuset~of~\\
~*~N~memory~nodes,~valid~values~for~n~are~0~..~N-1.~\\
~*/~\\
cpuset{\_}membind(cpuset{\_}p{\_}rel{\_}to{\_}sys{\_}mem(0,~n));
}\end{quote}
\end{quote}
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-export}{}
\pdfbookmark[1]{10.67~~~cpuset{\_}export}{cpuset-export}
\subsection*{10.67~~~cpuset{\_}export}


%___________________________________________________________________________

\hypertarget{int-cpuset-export-const-struct-cpuset-cp-char-buf-int-buflen}{}
\pdfbookmark[2]{int cpuset{\_}export(const struct cpuset *cp, char *buf, int buflen);}{int-cpuset-export-const-struct-cpuset-cp-char-buf-int-buflen}
\subsubsection*{\texttt{int cpuset{\_}export(const struct cpuset *cp, char *buf, int buflen);}}
\begin{quote}

Write the settings of cpuset \texttt{cp} to \texttt{file}.  If no file exists
at the path specified by \texttt{file}, create one.  If a file already
exists there, overwrite it.

Returns -1 and sets \texttt{errno} on error. Upon successful return, returns the
number of characters printed (not including the trailing '0' used to end
output to strings). The function cpuset{\_}export does not write more than size
bytes (including the trailing '0'). If the output was truncated due to this
limit then the return value is the number of characters (not including the
trailing '0') which would have been written to the final string if enough
space had been available. Thus, a return value of size or more means that
the output was truncated.

See \href{\#cpuset-text-format}{Cpuset Text Format} for details of the format of an exported
cpuset file.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-import}{}
\pdfbookmark[1]{10.68~~~cpuset{\_}import}{cpuset-import}
\subsection*{10.68~~~cpuset{\_}import}


%___________________________________________________________________________

\hypertarget{int-cpuset-import-struct-cpuset-cp-const-char-file-int-errlinenum-ptr-char-errmsg-bufptr-int-errmsg-buflen}{}
\pdfbookmark[2]{int cpuset{\_}import(struct cpuset *cp, const char *file, int *errlinenum{\_}ptr, char *errmsg{\_}bufptr, int errmsg{\_}buflen);}{int-cpuset-import-struct-cpuset-cp-const-char-file-int-errlinenum-ptr-char-errmsg-bufptr-int-errmsg-buflen}
\subsubsection*{\texttt{int cpuset{\_}import(struct cpuset *cp, const char *file, int *errlinenum{\_}ptr, char *errmsg{\_}bufptr, int errmsg{\_}buflen);}}
\begin{quote}

Read the settings of cpuset \texttt{cp} from \texttt{file}.  If no file exists
at the path specified by \texttt{file}, create one.  If a file already
exists there, overwrite it.

\texttt{Struct cpuset *cp} must have been returned by a previous \texttt{cpuset{\_}alloc()}
call. Any previous settings of \texttt{cp} are lost.

Returns 0 on success, or -1 on error, setting \texttt{errno}. Errors include
\texttt{file} not referencing a readable file.

If parsing errors are encountered reading the file, and if errlinenum{\_}ptr
is not NULL, then the number of the first line (numbers start with one)
with an error is written to *errlinenum{\_}ptr.  If an error occurs on the open,
and errlinenum{\_}ptr is not NULL, then zero is written to *errlinenum{\_}ptr.

If parsing errors are encountered reading the file and if errmsg{\_}bufptr
is not NULL, then it is presumed to point to a character buffer of at
least errmsg{\_}buflen characters, and a nul terminated error message is
written to *errmsg{\_}bufptr providing a human readable error message
explaining the error message in more detail.  As of this writing, the
possible error messages are:
\begin{quote}
\begin{itemize}
\item {} 
``Token 'CPU' requires list''

\item {} 
``Token 'MEM' requires list''

\item {} 
``Invalid list format: {\%}s''

\item {} 
``Unrecognized token: {\%}s''

\item {} 
``Insufficient memory''

\end{itemize}
\end{quote}

See \href{\#cpuset-text-format}{Cpuset Text Format} for details of the format required for imported
cpuset file.
\end{quote}


%___________________________________________________________________________

\hypertarget{cpuset-function}{}
\pdfbookmark[1]{10.69~~~cpuset{\_}function}{cpuset-function}
\subsection*{10.69~~~cpuset{\_}function}


%___________________________________________________________________________

\hypertarget{cpuset-function-const-char-function-name}{}
\pdfbookmark[2]{cpuset{\_}function(const char *function{\_}name);}{cpuset-function-const-char-function-name}
\subsubsection*{\texttt{cpuset{\_}function(const char *function{\_}name);}}
\begin{quote}

Return pointer to the named \texttt{libcpuset.so} function, or \texttt{NULL}. For base
functions that are in all implementations of libcpuset, there is no
particular value in using \texttt{cpuset{\_}function()} to obtain a pointer to the
function dynamically. But for \textbf{{[}optional]} cpuset functions, the use of
\texttt{cpuset{\_}function()} enables dynamically adapting to run-time environments that
may or may not support that function.

For more information, see the \titlereference{Extensible API} section, above.
\end{quote}


%___________________________________________________________________________

\hypertarget{system-error-numbers}{}
\pdfbookmark[0]{11~~~System Error Numbers}{system-error-numbers}
\section*{11~~~System Error Numbers}

The Linux kernel implementation of cpusets sets \texttt{errno} to specify the reason
for a failed system call affecting cpusets. These \texttt{errno} values are available
when a cpuset library call fails. Most of these values can also be displayed by
shell commands used to directly manipulate files below \texttt{/dev/cpuset}.

The possible \texttt{errno} settings and their meaning when set on a failed cpuset call
are as listed below.
\begin{quote}
\begin{description}
%[visit_definition_list_item]
\item[ENOSYS] %[visit_definition]

Invoked on an operating system kernel that does not support cpusets.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ENODEV] %[visit_definition]

Invoked on a system that supports
cpusets, but when the cpuset file system is not currently
mounted at
\texttt{/dev/cpuset}.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ENOMEM] %[visit_definition]

Insufficient memory is available.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EBUSY] %[visit_definition]

Attempted
\texttt{cpuset{\_}delete()}
on a cpuset with attached tasks.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EBUSY] %[visit_definition]

Attempted
\texttt{cpuset{\_}delete()}
on a cpuset with child cpusets.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ENOENT] %[visit_definition]

Attempted
\texttt{cpuset{\_}create()}
in a parent cpuset that doesn't exist.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EEXIST] %[visit_definition]

Attempted
\texttt{cpuset{\_}create()}
for a cpuset that already exists.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EEXIST] %[visit_definition]

Attempted to \texttt{rename(2)} a cpuset to a name that already exists.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ENOTDIR] %[visit_definition]

Attempted to \texttt{rename(2)} a non-existent cpuset.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[E2BIG] %[visit_definition]

Attempted a
\texttt{write(2)}
system  call on a special cpuset file
with a length larger than some kernel determined upper
limit on the length of such writes.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ESRCH] %[visit_definition]

Attempted to
\texttt{cpuset{\_}move()}
a non-existent task.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EACCES] %[visit_definition]

Attempted to
\texttt{cpuset{\_}move()}
a task which one lacks permission to move.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EACCES] %[visit_definition]

Attempted to \texttt{write(2)} a \texttt{memory{\_}pressure} file.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ENOSPC] %[visit_definition]

Attempted to \texttt{cpuset{\_}move()} a task to
an empty cpuset.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EINVAL] %[visit_definition]

The
\texttt{relcpu}
argument to
\texttt{cpuset{\_}pin()}
is out of range (not between
``zero'' and ``cpuset{\_}size() - 1'').

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EINVAL] %[visit_definition]

Attempted to change a cpuset in a way that would
violate a cpu{\_}exclusive or mem{\_}exclusive attribute
of that cpuset or any of its siblings.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EINVAL] %[visit_definition]

Attempted to write an empty cpus or mems bitmask
to the kernel.  The kernel creates new cpusets
(via mkdir) with empty cpus and mems, and the
user level cpuset and bitmask code works with
empty masks.  But the kernel will not allow an
empty bitmask (no bits set) to be written to the
special cpus or mems files of a cpuset.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EIO] %[visit_definition]

Attempted to
\texttt{write(2)}
a string to a cpuset tasks file that
does not begin with an ASCII decimal
integer.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EIO] %[visit_definition]

Attempted to \texttt{rename(2)} a cpuset outside of its current directory.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ENOSPC] %[visit_definition]

Attempted to \texttt{write(2)} a list to a cpus
file that did not include any online cpus.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ENOSPC] %[visit_definition]

Attempted to \texttt{write(2)} a list to a mems
file that did not include any online memory
nodes.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EACCES] %[visit_definition]

Attempted to add a cpu or mem to a cpuset that is
not already in its parent.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EACCES] %[visit_definition]

Attempted to set cpu{\_}exclusive or mem{\_}exclusive
on a cpuset whose parent lacks the same setting.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ENODEV] %[visit_definition]

The cpuset was removed by another task at the same
time as a \texttt{write(2)} was attempted on one of the
special files in the cpuset directory.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EBUSY] %[visit_definition]

Attempted to remove a cpu or mem from a cpuset
that is also in a child of that cpuset.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EFAULT] %[visit_definition]

Attempted to read or write a cpuset file using
a buffer that was outside your accessible address space.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ENAMETOOLONG] %[visit_definition]

Attempted to read a \texttt{/proc/<pid>/cpuset} file for
a cpuset path that was longer than the kernel page
size.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ENAMETOOLONG] %[visit_definition]

Attempted to create a cpuset whose base directory
name was longer than 255 characters.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ENAMETOOLONG] %[visit_definition]

Attempted to create a cpuset whose full pathname
including the ``/dev/cpuset'' is longer than 4095
characters.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ENXIO] %[visit_definition]

Attempted to create a \texttt{cpu{\_}exclusive} cpuset whose
\texttt{cpus} covered just part of one or more physical
processor packages, such as including just one of the
two Cores on a package.  For Linux kernel version 2.6.16
on i386 and x86{\_}64, this operation is rejected with this
error to avoid a fatal kernel bug.  Otherwise, this is a
normal and supported operation.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EINVAL] %[visit_definition]

Specified a cpus or mems list to the kernel which
included a range with the second number smaller than
the first number.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EINVAL] %[visit_definition]

Specified a cpus or mems list to the kernel which
included an invalid character in the string.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ERANGE] %[visit_definition]

Specified a cpus or mems list to the kernel which
included a number too large for the kernel to set
in its bitmasks.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ETIME] %[visit_definition]

Time limit for \href{\#cpuset-nuke}{cpuset{\_}nuke} operation reached without successful
completion of operation.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[ENOTEMPTY] %[visit_definition]

Tasks remain after multiple attempts by \href{\#cpuset-move-cpuset-tasks}{cpuset{\_}move{\_}cpuset{\_}tasks}
to move them to a different cpuset.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EPERM] %[visit_definition]

Lacked permission to kill (send a signal to) a task.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EPERM] %[visit_definition]

Lacked permission to read a cpuset or its files.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[EPERM] %[visit_definition]

Attempted to unlink a per-cpuset file.  Such files can not be
unlinked.  They can only be removed by removing (rmdir) the
directory representing the cpuset that contains these files.

%[depart_definition]
%[depart_definition_list_item]
\end{description}
\end{quote}


%___________________________________________________________________________

\hypertarget{change-history}{}
\pdfbookmark[0]{12~~~Change History}{change-history}
\section*{12~~~Change History}

Here is the history of changes to this document and associated software.
\begin{itemize}
\item {} 
2006-04-26  Paul Jackson  {\textless}\href{mailto:pj@sgi.com}{pj@sgi.com}{\textgreater}
\begin{quote}
\begin{itemize}
\item {} 
First published.

\end{itemize}
\end{quote}

\item {} 
2006-11-14  Paul Jackson  {\textless}\href{mailto:pj@sgi.com}{pj@sgi.com}{\textgreater} Version 1 (\href{\#cpuset-version}{cpuset{\_}version}())
\begin{quote}
\begin{itemize}
\item {} 
Added \href{\#cpuset-nuke}{cpuset{\_}nuke} routine.

\item {} 
Added various \texttt{cpuset{\_}fts{\_}*} routines.

\item {} 
Added various \texttt{cpuset{\_}*{\_}affinity} routines.

\item {} 
Added \href{\#cpuset-move-cpuset-tasks}{cpuset{\_}move{\_}cpuset{\_}tasks} routine.

\item {} 
Converted from using \texttt{ftw(3)} to \texttt{fts(3)}.

\item {} 
Various minor documentation errors fixed.

\item {} 
Added \href{\#cpuset-version}{cpuset{\_}version} routine.

\item {} 
Improved error checking by \href{\#cpuset-move-all}{cpuset{\_}move{\_}all}.

\item {} 
Correct system call numbering for sched{\_}setaffinity on i386 arch.

\item {} 
Correct \href{\#cpuset-latestcpu}{cpuset{\_}latestcpu} result if command basename has space character.

\item {} 
Remove 256 byte limit on \href{\#cpuset-export}{cpuset{\_}export} output.

\end{itemize}
\end{quote}

\item {} 
2007-01-14  Paul Jackson {\textless}\href{mailto:pj@sgi.com}{pj@sgi.com}{\textgreater} Version 2 (\href{\#cpuset-version}{cpuset{\_}version}())
\begin{quote}
\begin{itemize}
\item {} 
Fix \href{\#cpuset-create}{cpuset{\_}create}, \href{\#cpuset-modify}{cpuset{\_}modify} to not zero undefined attributes
such as \texttt{memory{\_}spread{\_}page}, \texttt{memory{\_}spread{\_}slab} and
\texttt{notify{\_}on{\_}release}. See further the explanation of an ``\emph{undefined}
mark'', in \href{\#cpuset-programming-model}{Cpuset Programming Model}. \href{\#cpuset-create}{cpuset{\_}create} now respects
default kernel cpuset creation settings, and \href{\#cpuset-modify}{cpuset{\_}modify} now respects
the existing settings of the cpuset, unless explicitly set.

\end{itemize}
\end{quote}

\item {} 
2007-04-01  Paul Jackson {\textless}\href{mailto:pj@sgi.com}{pj@sgi.com}{\textgreater} Version 3 (\href{\#cpuset-version}{cpuset{\_}version}())
\begin{quote}
\begin{itemize}
\item {} 
Fix \href{\#cpuset-setcpus}{cpuset{\_}setcpus}() and \href{\#cpuset-setmems}{cpuset{\_}setmems}() to mark the
\texttt{cpus} and \texttt{mems} fields as defined, so that setting them
before doing a \href{\#cpuset-create}{cpuset{\_}create}() has affect.

\end{itemize}
\end{quote}

\end{itemize}

\end{document}
