CDC.IO Setup for Developers

The CDC.IO Project is a host-based, command-line utility which enables manipulation of dtcyber NOS (1 and 2) PLATO masterfiles and containers.

License

GNU General Public License

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Release 2.3.1

The first release of the utility on July 4, 2023 (US Independence Day) is significant in that it represents some independence from the tyranny of PLATO files having to be managed through the operating system vehicles of “virtual tape”.

Source Availability

Access to the sources is currently constrained to members of the Cybis Development group at Retro1.Org. Documentation is available but until a rigorous testing all functionality can be done, the sources will remain constrained.

Documentation

The SPHINX documentation system is used to maintain the package documentation.

The output of SPHINX is periodically updated at https://static.retro1.org/cdc.io.

Installation

Install GIT

It is assumed that you have GIT installed. There are numerous references available on the web.

Install Python

With the PYTHON interpreter installed from Python.Org

Choose a suitable location (workspace) and clone the cdc.io repository. In my example, I'll use the directory rooted at E:\Engineering\SZ10101.

In the following examples - I'm using Cmder as my console application. It understands both windows and linux shell commands (generally).

Clone the Repository

Open a command prompt and issue the following commands:

E:\Engineering\SZ10101 λ git clone "https://github.com/retro1-org/cdc.io" cdc.io.test Cloning into 'cdc.io.test'... remote: Enumerating objects: 228, done. remote: Counting objects: 100% (228/228), done. remote: Compressing objects: 100% (134/134), done. Receiving objects: 43% (99/228)used 228 (delta 67), pack-reused 0 Receiving objects: 100% (228/228), 568.99 KiB | 5.69 MiB/s, done. Resolving deltas: 100% (67/67), done.

Verify Python

Now, make sure you have properly configured python.

E:\Engineering\SZ10101 λ python --version Python 3.11.5 E:\Engineering\SZ10101 λ pip install virtualenv

The virtualenv (virtual environment) command will be installed… but if you see Requirement already satisfied messages, that's fine too. You're good-to-go!

Create the Virtual Environment

Now set up the necessary environment for the “sandboxed” python interpreter.

The first step is to create a virtual environment, then activate it.

E:\Engineering\SZ10101 λ cd cdc.io.test E:\Engineering\SZ10101\cdc.io.test (main -> origin) λ virtualenv .\py11.venv created virtual environment CPython3.11.5.final.0-64 in 469ms creator CPython3Windows(dest=E:\Engineering\SZ10101\cdc.io.test\py11.venv, clear=False, no_vcs_ignore=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\sjzop\AppData\Local\pypa\virtualenv) added seed packages: pip==23.2.1, setuptools==68.2.2, wheel==0.41.2 activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator E:\Engineering\SZ10101\cdc.io.test (main -> origin) λ .\py11.venv\Scripts\activate.bat E:\Engineering\SZ10101\cdc.io.test (main -> origin) (py11.venv) λ

Install CDC.IO

Once the virtual environment has been installed and activated, we can now install cdc.io into the environment using the following commands. First, cd cdc.io to ensure that you're in the proper place from which the installation can take place.

E:\Engineering\SZ10101\cdc.io.test (main -> origin) (py11.venv) λ cd cdc.io E:\Engineering\SZ10101\cdc.io.test\cdc.io (main -> origin) (py11.venv) λ pip install . Processing e:\engineering\sz10101\cdc.io\cdc.io Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Requirement already satisfied: ansimarkup in e:\engineering\sz10101\py11.venv\lib\site-packages (from cdc.io==2.2.1) (2.1.0) Requirement already satisfied: colorama in e:\engineering\sz10101\py11.venv\lib\site-packages (from cdc.io==2.2.1) (0.4.6) Requirement already satisfied: pandas in e:\engineering\sz10101\py11.venv\lib\site-packages (from cdc.io==2.2.1) (2.1.1) Requirement already satisfied: numpy>=1.23.2 in e:\engineering\sz10101\py11.venv\lib\site-packages (from pandas->cdc.io==2.2.1) (1.26.0) Requirement already satisfied: python-dateutil>=2.8.2 in e:\engineering\sz10101\py11.venv\lib\site-packages (from pandas->cdc.io==2.2.1) (2.8.2) Requirement already satisfied: pytz>=2020.1 in e:\engineering\sz10101\py11.venv\lib\site-packages (from pandas->cdc.io==2.2.1) (2023.3.post1) Requirement already satisfied: tzdata>=2022.1 in e:\engineering\sz10101\py11.venv\lib\site-packages (from pandas->cdc.io==2.2.1) (2023.3) Requirement already satisfied: six>=1.5 in e:\engineering\sz10101\py11.venv\lib\site-packages (from python-dateutil>=2.8.2->pandas->cdc.io==2.2.1) (1.16.0) Building wheels for collected packages: cdc.io Building wheel for cdc.io (pyproject.toml) ... done Created wheel for cdc.io: filename=cdc.io-2.2.1-py3-none-any.whl size=590499 sha256=730a28a27ca8582ac5f38057f09c298c77cd8a21908ba38e2f48c8800e728a53 Stored in directory: C:\Users\sjzop\AppData\Local\Temp\pip-ephem-wheel-cache-3md12bae\wheels\14\5b\1c\cb3bb4f2ad2067e7b92e77f8a3c89196982aba498b6e49572f Successfully built cdc.io Installing collected packages: cdc.io Attempting uninstall: cdc.io Found existing installation: cdc.io 2.2.1 Uninstalling cdc.io-2.2.1: Successfully uninstalled cdc.io-2.2.1 Successfully installed cdc.io-2.2.1 E:\Engineering\SZ10101\cdc.io.test\cdc.io (main -> origin) (py11.venv) λ

At this point - all of the dependencies have been resolved and there remains one additional housekeeping step. One of the more annoying features of windows is that the CMD terminal runs with the Windows codepage (Default is Windows 437). Python (and specifically CDC.IO expects stdin, stdout and stderr to handle UTF-8 encoding for redirection and display.

Running cdc.io –help reveals this housekeeping step (highlighted in GREEN):

E:\Engineering\SZ10101\cdc.io.test\cdc.io (main -> origin) (py11.venv) λ cdc.io --help (Module) 'cdc.io' Completed. E:\Engineering\SZ10101\cdc.io\cdc.io\dist>cdc.io -h (Module) 'cdc.io' Started. You may need to set the environment variable using the command: set PYTHONIOENCODING=UTF-8 # Windows (or) set PYTHONUTF8=1 # Windows export PYTHONUTF8=1 # Linux / MacOS in order to redirect UTF-8 output. usage: cdc.io [-h] {allhelp,banner,c.aclist,c.analyze,c.b66conv,c.backup,c.brestore,c.catalog,c.findnf,c.findpf,c.grplist,c.libget,c.nosdump,c.nosget,c.pfcopy,c.pflist,c.pfprint,c.resolve,cat.compare,cat.delete,cat.list,dc.pack,dc.unpack,hash,noslist,pfget,pflist,pfprint,version} ... Perform PLATO File, NOS Pack and PLATO Pack manipulation and display. options: -h, --help show this help message and exit subcommands: valid subcommands {allhelp,banner,c.aclist,c.analyze,c.b66conv,c.backup,c.brestore,c.catalog,c.findnf,c.findpf,c.grplist,c.libget,c.nosdump,c.nosget,c.pfcopy,c.pflist,c.pfprint,c.resolve,cat.compare,cat.delete,cat.list,dc.pack,dc.unpack,hash,noslist,pfget,pflist,pfprint,version} additional help allhelp allhelp -h banner banner -h c.aclist c.aclist -h c.analyze c.analyze -h c.b66conv c.b66conv -h c.backup c.backup -h c.brestore c.brestore -h c.catalog c.catalog -h c.findnf c.findnf -h c.findpf c.findpf -h c.grplist c.grplist -h c.libget c.libget -h c.nosdump c.nosdump -h c.nosget c.nosget -h c.pfcopy c.pfcopy -h c.pflist c.pflist -h c.pfprint c.pfprint -h c.resolve c.resolve -h cat.compare cat.compare -h cat.delete cat.delete -h cat.list cat.list -h dc.pack dc.pack -h dc.unpack dc.unpack -h hash hash -h noslist noslist -h pfget pfget -h pflist pflist -h pfprint pfprint -h version version -h Copyright (C) 2021-2024 Steve Zoppi (GitHub:sz10101) Details ------------------------------------------------------------------------ Unless otherwise indicated - all switches are optional with the defaults set as indicated above. In general, commands operate on one of four types of virtual disk pack (container) objects: PLATO A virtual device container which represents a Plato Disk Device. This type of container is free of any other operating system artifacts (as are found on NOS(1 or 2) disk packs) and is assumed to have a device geometry of an 844-type pack. NOS A virtual device container which represents a properly formatted NOS1 or NOS2 pack. NovaNET A host-based filesystem hierarchy which contains files in the format required by the NovaNET system. Backup A host-based filesystem hierarchy which contains backup files written by this utility. Backup files may be restored to any supported device type. The backup hierarchy resembles, but is NOT the same as the NovaNET storage format. ** NOTE ** This utility currently operates on NOS Pack formats and correctly interprets label Level 0 and 2. The following pack geometries are automatically detected: DI (844-21) DJ (844-41/Half-Track) DL (844-41/Full-Track) DK (7152/7154/844-21/Full-Track) DM (885) DQ (885) DB (885-42) Although most FILE manipulations may be performed on individual packs, the NOS permanent file system enables the grouping of packs into "families" which must all be concurrently present when file operations are performed on pack "families" which have more than one member. When a pack family has more than one member, the device number is recognized but no operations may be performed without the member which holds the family's catalog. (Direct and Indirect access files are fully-supported). To facilitate "family" operations, and to enable bulk operations that span more than one pack of any type, the concept of a "cluster" is a central concept in the context of this utility. About Clusters -------------- A cluster is (conceptually) a logical group of packs which have a relationship to each other in a system's running context. A cluster "collection" is automatically discovered by scanning the contents of the path specified by the "-c" or "--cluster" command line option. If the "cluster" option targets a file, that file is interpreted as as Cluster Definition File (example above). This file associates a device number with a given virtual container. When "cluster" targets a path, ALL files stored at that path are automatically examined and, if all automatic tests are satisfied, the pack is added to the cluster. In the case of a NOS2 disk structure which contains PLATO Packs, all PLATO Master files are automatically sought and loaded into the cluster. Clusters are useful when searching for, or manipulating files across multiple container files. This is especially helpful when only the filename is known (frequently the case with a PLATO system). The Cluster Path Specification ------------------------------ When specifying clusters the syntax is generally: <path>[;<user_index>] where: <path> is the path to either a file or a directory. If the path contains spaces, it must be encapsulated in quotation-marks. <user_index> (only applicable to NOS Pack Structures) is the User Index under which the Master Files are cataloged. This specifier is ignored in all other instances. The Cluster Object Specification -------------------------------- A cluster object is one which depends on ALL accounts and packs participating in a running system, to be together so a function can be performed across them. The syntax for a cluster object is:: [<account.>]filename[@[<packname>[.][<(-|+)size>]]] Required: filename: First found filename in the cluster Optional: account: For "Target" specifiers: The name of the account receiving the file For "Source" specifiers: Ignored. packname: pack upon which this file SHOULD be found or will be placed. size: for some operations, indicates the number of parts. For size change operations, the "+" plus or "-" minus indicators are required. The default specifiers for unsupplied nodes are: account = "*" (any) packname = "*" (any) size = "*" (existing) Detailed Documentation -------------------------------------------------------- Additional information and detailed explanations of all subcommands may be found at: https://codex.retro1.org/plato:cdc.io:start End of Function. (Module) 'cdc.io' Completed.

Now issue the set PYTHONIOENCODING=UTF-8 command and now, you should be able to use all of the functionality cdc.io has to offer!

Continue to Learn

Continue to learn on the Retro1.Org CODEX

Retro1.org codex

Table of Contents