Roles Database Data Feed Programs
Last modified 86/1004/20092010
Connecting to the Roles production or test server
...
On the production Roles server (roles.mit.edu, aka cloverleaf) and the test Roles server (roles-test.mit.edu, aka parsley), there are several sets of data feed jobs that are automatically run each day. The overall schedule can be found in the crontab file:
Production: ~rolesdb/cronjobs/crontab.cloverleaf
Test: ~rolesdb/cronjobs/crontab.parsley
If you ever need to change the crontab file, do the following: 1.
- Connect to the server machine using telnet or SSH as user rolesdb.
...
- Go to the cronjobs directory
cd cronjobs
...
- Check out the crontab file from RCS using the alias "checkout"
checkout crontab.cloverleaf
(or)
checkout crontab.parsley
...
- Use emacs or vi to make the desired changes
...
- Check in the new crontab file into RCS using the alias "checkin"
checkin crontab.cloverleaf
(or)
checkin crontab.parsley
...
- ***
...
- Don't forget this step! **
...
- Run the crontab command.
crontab crontab.cloverleaf
(or)
crontab crontab.parsley
You can display the current crontab entry with the command crontab -l
...
Each crontab file runs several shell scripts that, in turn, run individual programs. These high-level shell scripts include:
morning_jobs_early | - Extracts most data from the Warehouse, including |
...
most types of Qualifiers, and does some processing |
...
for the Master Department Hierarchy |
morning_jobs_late - Runs some steps that depend on PERSON data from
the Warehouse including (a) loading the PERSON table
from krb_person@warehouse, (b) loading EHS-related
Room Set data from the Warehouse into RSET
qualifiers, (c) processing externally-derived
authorizations (table EXTERNAL_AUTH).
(Runs in the morning, not quite so early)
...
Directory | Description |
|---|---|
~rolesdb/archive | Some compressed historical files from previous days' data feed runs |
~rolesdb/bin | Generic data feed perl scripts and other program files |
~rolesdb/bin/ehs | EHS-related data feed programs |
~rolesdb/bin/extract | Programs related to out-going data for DACCA, LDS (SAP component being phased out), Budget (failed SAP project), etc. |
~rolesdb/bin/pdorg | Programs related to out-going data for updating PD Org entries in SAP related to APPROVER authorizations |
~rolesdb/bin/repa_feed | Temporary or test versions of programs |
~rolesdb/bin/roles_feed | Most data feed programs for data coming into the Roles DB |
~rolesdb/data | Data files used by data feed programs. Most data files are temporary, but some, such as roles_person_extra.dat, are permanent. |
~rolesdb/doc | Miscellaneous notes and documentation |
~rolesdb/extract | Empty |
~rolesdb/lib | A few generic perl modules, and some config files |
~rolesdb/log | Most recent log files from data feed programs |
~rolesdb/misc | Miscellaneous notes and working files |
~rolesdb/sap_feed | Obsolete versions of Roles->SAP data feed programs |
~rolesdb/sql | SQL source files for creating tables, views, and stored procedures. Files for creating tables (new_schema*.sql) are preserved for documentation purposes and should NOT be rerun -- tables should never be dropped and recreated since we do not want to lose the data. Files for creating stored procedures and views can be modified and rerun. |
~rolesdb/sql/frequently_run_scripts | Special SQL scripts that are run periodically, e.g., to analyze tables |
~rolesdb/sap_feed | Obsolete versions of Roles->SAP data feed programs |
~rolesdb/xsap_feed/bin | Programs for Roles->SAP data feed programs |
~rolesdb/xsap_feed/config | Config files for Roles->SAP data feed programs |
~rolesdb/xsap_feed/data | Nightly data for Roles->SAP data feed programs |
...