Thursday, 16 November 2017

Ansible: How installing a group of packages?

Installing a set of packages with Ansible

The context

I need to install a group of packages on multiple Linux servers. Ansible is the right tool for that.


The host file: hosts

[test_machines]
   test1.my_domain.com
   test2.my_domain.com
[dev]
   dev1.my_domain.com
   dev2.my_domain.com
[uat]
   uat1.my_domain.com
   uat2.my_domain.com
[prod]
   prd1.my_domain.com
   prd2.my_domain.com


The playbook : add_packages.yml

- hosts: all
  vars_files:
    - data/all_my_packages.yml

  tasks:
  - name: Ensure a set of packages is installed.
    yum:
      name={{ item.name }}
      state={{ item.state }}
    with_items: '{{ packages }}'



The datafile: data/all_my_packages.yml

packages:
 - { name: 'glibc.i686', state: 'present' }
 - { name: 'glibc.x86_64', state: 'present' }
 - { name: 'glibc-common', state: 'present' }
 - { name: 'glibc-devel', state: 'present' }
 - { name: 'glibc-headers', state: 'present' }
 - { name: 'libstdc++.i686', state: 'present' }
 - { name: 'libstdc++.x86_64', state: 'present' }
 - { name: 'libstdc++-devel.x86_64', state: 'present' }
 - { name: 'compat-libstdc++-33.i686', state: 'present' }
 - { name: 'compat-libstdc++-33.x86_64', state: 'present' }
 - { name: 'compat-libstdc++-296-2.96-144.el6.i686', state: 'present' }
 - { name: 'pam.i686', state: 'present' }
 - { name: 'pam.x86_64', state: 'present' }
 - { name: 'gtk2.i686', state: 'present' }
 - { name: 'gtk2.x86_64', state: 'present' }
 - { name: 'xorg-x11-xauth.x86_64', state: 'present' }
 - { name: 'dos2unix', state: 'present' }



The command

Checking on subset of servers:
# ansible-playbook add_packages.yml --check -i hosts --limit test_machines


Running:
# ansible-playbook add_packages.yml -i hosts

Friday, 20 October 2017

The context

For a project, we had to create over 100 virtual machines (VM), each one with specific caracteristics: OS, # CPU, RAM, # disks, disk space... About 1/3 VM are Windows, 2/3 Linux.
I maintain a MS Access Database with the requested VM.
The project is near ending; for quality control, I need to validate what is delivered vs. what was requested. My sysadmins went on almost(*) all the VM to get the caracteristics; the result is 2 text files, one for Windows and one for Linux.
*: This is important. I don't have all the VM yet, hence I will have to do this process again; this have to be taken into account in my solution.

How to compare the data?

Method 1: by hand

Boring, tedious and highly error prone. Forgotten.

Method 2: using Excel

I tried that first. But I'm not a king on Excel, and because the caracteristics are quite different between Linux and Windows, is't hard to make comparisons. Abandoned.

Method 3: using Access

This is actually the most logical way, because my initial source of information is already in Access. Moreover, I do SQL for many years, so I feel comfortable to compare the servers with it (despite Access's SQL is somehow different from standard).
However, because the caracteristics are different between each server, a import from the text file did not yield good results.
But wait! Access is able to manage XML... let's try that.

What I've done?

  1. I've merged the 2 text files
  2. I've transform the raw text file into XML with a basic text editor; tedious, and I made a lot of errors, but with the help of online XML syntax checker, I've ended with a clean file.
  3. Import the XML to Access: encountered a first problem
Problem #1: attributes are badly managed
My XML file was similar to this:
<servers>
  <server kernel="..." name="..." os="..." ..>
    <filesystems>
       <fs mount="/dev" name="devtmpfs" size="5.8G" ... >
       ...
    </filesystems>
   </server>
   ...
</servers>
However, using attributes was a bad idea: Access doesn't seem able to manage them and I ended with empty fields in the result table:
To overcome this, I turned the attributes into elements (easy, using the Find & Replace function of Notepad):
<servers>
 <server> <name>...</name> <os>...</os> <ci>...</ci> ...
   <filesystems>
     <fs> <name>devtmpfs</name> <size>12G</size> <mount>/dev</mount> ... </fs>
     ....
   </filesystems>
 </server>
 ....
<servers>
Problem #2: no common key between the generated tables
Access ended with 2 tables: server and filesystems. However, no primary key is defined in server, but worse no foreign key is defined in filesystems!
The solution: transforming the XML during the import. Access allows to specify a XSLT file to transform the XML.
XML source --> XSLT transformation --> Access tables
Here is my XSLT:
<xml version="1.0" encoding="UTF-8"?>
  <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output indent="yes" omit-xml-declaration="yes" />
    <xsl:strip-space elements="*" />
    <xsl:template match="node()">
      <xsl:copy>
        <xsl:apply-templates select="node()">
      </xsl:copy>
    </xsl:template>
    <xsl:template match="fs">
      <fs>
        <ci><xsl:apply-templates select="../ci"></ci>
      </fs>
    </xsl:template>
  </xsl:stylesheet >

Update: gathering data with Ansible

I now have access to an Ansible console; this will allow me to gather the data on a regular basis. Sadly, Ansible output format is JSON, and not XML. However, JSON can easily be transformed to XML with online tools such as https://www.freeformatter.com/json-to-xml-converter.html

Thursday, 12 October 2017

MS Access: embedded INNER JOIN

How to embed INNER JOIN in MS Access:


SELECT ...
FROM ( ( T1  INNER JOIN  T2 ON ... )

       INNER JOIN  T3 ON ... )
     INNER JOIN T4  ON ...;

Friday, 6 October 2017

Roland TR77

A new rhythm box in my studio: Roland TR77. Released in 1972 but sold alonside the 70's , mine was built in November 1976.

Many latin rhythms: Chacha, Beguine, Mambo, Samba,  Rumba, Bossa Nova, ... Very good sound, a lot of modulation. A very interesting instrument.
The Mambo could be heard, among many Polymoog filter sweeps, in one of my favorite song: "Song For Guy" by Elton John.

Saturday, 23 September 2017

Emulation of Fujitsu MB8877 Floppy Disk Controler with Arduino


Context

I have a Yamaha QX1 MIDI sequencer from 1984.
The backup medium is 5"1/4 floppy disks.
I want to replace the floppy drive (Canon MF-221) and the Floppy Disk Controler (Fujitsu MB8877a) with an Arduino.

The project

Description

The Fujitsu MB8877 is a second source for Western Digital FD1793.
(white pins are not connected on the QX1 DM board)




 On the QX1:

  • Pin 37 (/DDEN) is pulled down as we only use Double Density disks (256 bytes/sector).
  • Pin 3 (/CS) is grounded via a 220 Ohms resistor (always selected).
  • Pin 24 (Clock) is only used to time the FDC logic; we don't use this signal to clock the Arduino. However, the clock starts only ~5.4 msec after powering up the QX1; before that, all signals are garbage.
  • Pin 19 (Master Reset) is used on boot. MR goes low ~4.6 msec after power up (the main CPU Hitachi HD68B09 only needs 110 nsec @ 8 MHz to boot).
  • Pins 7 to 14: Data bus. This bus is shared by several ICs on the board: 
    • IC27: Fujitsu MB8877, the floppy disk controller
    • IC8: Yamaha YM5208, an ASIC named PMAC (seems to be a DMA)
    • IC31: Yamaha YM5213, an ASIC named PIT (interface with PN board: display, keyboard, ...)
    • IC26: Yamaha YM5212, an ASIC named MASTER (UART and MIDI interface)
    • IC7: a buffer (74LS245) to the main CPU
    • IC17: a buffer (74LS244) to the RAM
    • IC23, 24 and 25: the ROMs
  • Pins 5 and 6 (A0, A1): selects register (cf table below)
  • Pins 2 and 4 (/WE, /RE): selects function (cf table below)
  • Pin 39 (IRQ): allows Arduino to warns QX1 the previous command is completed
  • Pin 38 (DRQ): allows Arduino to warns QX1 data is available (read) or is required (write) on DAL0-7.
The following signals are normally used to drive the Disk Drive. However, on the QX1, these signals are captured by IC8 (PMAC).  My guess is PMAC is also in charge of the precise timing for pre-comp.
  • Pin 17 (Early) and 18 (Late): pre-compensation signals. These signals are used to shift the timing of data to be written on the disk. 0=No pre-comp, 1=Pre-comp.
  • Pin 29 (TG43): warns the CPU on head position. 0=head on track 0 to 43, 1=head on track 44 to 76.
  • Pin 31 (WD): Data to be written on the disk drive

The protocol

The protocol between the MPU and the FDC is driven by the MPU. It basically consists for the MPU to read or post data in the FDC's registers. To do so, the MPU controls 5 pins: /CS (to select the FDC), /RE and /WE (to select read or write), A1 and A0 (to select the register).
As mentioned earlier, /CS is grounded: the chip is always selected.
Thus, the control is ensured by 4 pins, summarized in this table:

/CS A1 A0 /RE /WE
Function
0 0 0 0 1
MPU wants to read Status Register
0 0 1 0 1
MPU wants to read Track Register
0 1 0 0 1
MPU wants to read Sector Register
0 1 1 0 1
MPU wants to read Data Register
0 0 0 1 0
MPU wants to write into Command Register
0 0 1 1 0
MPU wants to write into Track Register
0 1 0 1 0 MPU wants to write into Sector Register
0 1 1 1 0
MPU wants to write into Data Register
0 - - 0 0
Not specified in the datasheet: see below
The situation where both /WE (MPU want to read a register) and /RE (MPU wants to post to a register) are low simultaneously is not specified in the datasheet, and doesn't make sense. However, I measured a lot of cases where both /WE and /RE are down.

Schematic

The schematic can be found here: Project schematic (PDF) 

Some explanations

U1 (74LS245): buffer with DAL0-7

U3 (74LS244): buffer from the keyboard and the command pins (A0, A1, /RE and /WE)

U3 (74LS244): buffer to the LCD

U7 (74HC125): voltage adaptor (5V -> 3.3V) for the SD card and reverse voltage for the keyboard.

U4A (74LS139): this component manage the 8 bit bus (port D on the Arduino) by enabling/disabling the buffers U1, U3 and U6. U4A is managed via port C with the following table:
C3 C2 C1 C0 D7 D6 D5 D4 D3 D2 D1 D0 Sens Description
1 - - - - - - - - - - - - High Z
0 - 1 1 DB7 DB6 DB5 DB4 - R/W E - Output LCD
0 - 1 0 << < > >> - - - - Input Keyboard
0 - 0 1 - - A1 A0 /WE /RE - - Input Command
0 0 0 0 DAL7 DAL6 DAL5 DAL4 DAL3 DAL2 DAL1 DAL0 Input Data
0 1 0 0 DAL7 DAL6 DAL5 DAL4 DAL3 DAL2 DAL1 DAL0 Output Data


The QX1 - timings

All the timings below are measured with a Saleae Logic 8 Analyzer at 12 Ms/Sec.

Bootup sequence

Here are the chronograms (timing) of the QX1 bootup sequence.





Problem with /RE and /WE

On the MB8877 datasheet, no information if both are low. However, I found many situation where /RE and /WE are both down.

Examples where /RE and /WE are both low, an undefined situation.

Timing for the read operation

The timing for a read operation (the MPU wants to read a register of the FDC) is the following:



Wednesday, 16 August 2017

MS Access: fancy InStr function

MS Access is a pretty good database included within Office Pro.


The problem
I have tables with a list of Linux packages installed on several servers, one table per server. I want a table with all the packages common to all servers, and for each package (ex: audit-libs-python-1.8-2.el5), the package root (audit-libs-python) and the version (1.8-2.el5).


Step 1: selecting the common packages


SELECT DISTINCT Srv1.Package
FROM Srv1, Srv2, Srv3, Srv4
WHERE Srv1.Package = Srv2.Package
and Srv2.Package = Srv3.Package
and Srv3.Package = Srv4.Package;



Step 2: extracting the package root and version
Package versions are not always written in a similar way. Some examples:
Package                Root        Version
anacron-2.3-45.el5     anacron     2.3-45.el5
aspell-en-6.0-3        aspell-en   6.0-3
bzip2-1.0.3-6.el5_5    bzip2       1.0.3-6.el5_5
db4-4.3.29-10.el5_5.2  db4         4.3.29-10.el5_5.2


However, the version starts with an hyphen (-) followed by a digit (0 to 9). Unfortunately, MS Access doesn't allow the use of Regex (regular expressions).
To overcome this, I replaced all digits by a special character (#) and then searched for the position of "-#"; the substring before this position is the package root, the one after is the version.


SELECT Package,
left(Package,instr(Package,"-")+
 instr(
  Replace(
   Replace(
    Replace(
     Replace(
      Replace(
       Replace(
        Replace(
         Replace(
          Replace(
           Replace(
mid(Package,instr(Package,"-")),"9","#"),
          "8","#"),
         "7","#"),
        "6","#"),
       "5","#"),
      "4","#"),
     "3","#"),
    "2","#"),
   "1","#"),
  "0","#"),
 "-#")-2) as PackageRoot,

mid(Package,instr(Package,"-")+
 instr(
  Replace(
   Replace(
    Replace(
     Replace(
      Replace(
       Replace(
        Replace(
         Replace(
          Replace(
           Replace(mid(Package,instr(Package,"-")),"9","#"),
          "8","#"),
         "7","#"),
        "6","#"),
       "5","#"),
      "4","#"),
     "3","#"),
    "2","#"),
   "1","#"),
  "0","#"), "-#")) as PackageVersion


FROM [Common packages];


Sure that writing a VBA function might do the trick as well, but I wanted a SQL version.


Wednesday, 9 August 2017

The 2044 Is Back

Long time ago, the SSM 2044 4-Pole Voltage Controlled Filter was everywhere. The chip was designed by Dave Rossum (E-MU).
Its characteristic sound could be heard from different synths: Crumar Bit 01, most EMU products, Korg Mono/Poly, Polysix and Trident (Mk I and Mk II), Octave Plateau Voyetra 8 (used by Eurythmics), PPG Wave (used by Tangerine Dream) and many others.


However, Solid State Micro Technology for Music is long ago gone, and SSM 2044 are now rare to find - mostly from dismantled synths - and, of course, expensive.


Hopefully, Dan Parks, a former SSM employee, has decided to recreate the legendary filter with his company, Sound Semiconductor.
The new filter is named SSI2144, and the datasheet is here: http://soundsemiconductor.com/downloads/ssi2144datasheet.pdf