I/O Hardware  Incredible variety of I/O devices  Common concepts  Port  Bus (daisy chain or shared direct access)  Controller (host adapter)  I/O instructions control devices  Devices have addresses, used by  Direct I/O instructions (the controller has one or more registers for data and control signals).  Memory-mapped I/O (device control registers are mapped into the address space of the processor). Operating System Concepts with Java – 8th Edition 12.25 Silberschatz, Galvin and Gagne ©2009
View full slide show




I/O Hardware   Incredible variety of I/O devices  Storage  Transmission  Human-interface Common concepts – signals from I/O devices interface with computer  Port – connection point for device  Bus - daisy chain or shared direct access   PCI bus common in PCs and servers, PCI Express (PCIe)  expansion bus connects relatively slow devices Controller (host adapter) – electronics that operate port, bus, device  Sometimes integrated  Sometimes separate circuit board (host adapter)  Contains processor, microcode, private memory, bus controller, etc – Some talk to per-device controller with bus controller, microcode, memory, etc Operating System Concepts – 9th Edition 13.5 Silberschatz, Galvin and Gagne ©2013
View full slide show




Introduction to the new mainframe Facilities supplied by Cross Memory (XMS) • Data movement between address spaces - several instructions are provided which can perform this function. MVCP moves data from the secondary address space to the primary; MVCS moves data in the opposite direction; and MVCK performs a similar function between areas with different storage protect keys. Note that these instructions require that the function knows the addresses of the data areas it wishes to use in both address spaces - finding these can be complicated! PAS SAS 1. Program sharing - the PC (program call) instruction can be used to transfer control to a program in another address space, and the PT (program transfer) or PR (program return) instructions used to return. This requires the function running in the original address space to know how to find the program to be called, and Cross Memory Services provides a control block structure containing "linkage tables" and "entry tables" to hold this information. • PAS SAS 2. PAS SAS PAS = Primary Address Space Address Space 3. SAS = Secondary • Data sharing between address spaces - address spaces can be set up which act as data servers for a number of other address spaces, then one of several methods used to access the data from the "clients", without actually moving it into the client address space. For example, programs running in common storage can switch into secondary mode to access data in the shared address space (in primary mode, ordinary machine instructions are taken to refer to storage in the primary address space, but in secondary mode they are taken to refer to storage in the secondary address space). Alternatively, the program call mechanism (see previous bullet point) can be used to invoke a "server" program in the shared address space to access data and return a result to the client. ( Used for XMS Control Block and Structures such as linkage and entry tables). Note: Second address space initialized SDSF © Copyright IBM Corp., 2010. All rights reserved. Page 53 of 85
View full slide show




Introduction to the new mainframe Storage areas in an address space z/OS V1R13 BAR Problem (user) programs Run here LINE CVT (offset 16 (hex10) within PSA) All storage above 2 GB This area is called high virtual storage and is addressable only by programs running in 64-bit mode. It is divided by the high virtual shared area, which is an area of installation-defined size that can be used to establish cross-address space viewable connections to obtained areas within this area. Extended areas above 16 MB This range of areas, which lies above the line (16 MB) but below the bar (2 GB), is a kind of “mirror image” of the common area below 16 MB. They have the same attributes as their equivalent areas below the line, but because of the additional storage above the line, their sizes are much larger. Nucleus This is a key 0, read-only area of common storage that contains operating system control programs. System queue area (SQA) (2048 MBs)This area contains system level (key 0) data accessed by multiple address spaces. The SQA area is not pageable (fixed), which means that it resides in central storage until it is freed by the requesting program. The size of the SQA area is predefined by the installation and cannot change while the operating system is active. Yet it has the unique ability to “overflow” into the CSA area as long as there is unused CSA storage that can be converted to SQA. Pageable link pack area (PLPA), fixed link pack area (FLPA), and modified link pack area (MLPA) This area contains the link pack areas (the pageable link pack area, fixed link pack area, and modified link pack area), which contain system level programs that are often run by multiple address spaces. For this reason, the link pack areas reside in the common area that is addressable by every address space, therefore eliminating the need for each address space to have its own copy of the program. This storage area is below the line and is therefore addressable by programs running in 24-bit mode. CSA This portion of common area storage (addressable by all address spaces) is available to all applications. The CSA is often used to contain data frequently accessed by multiple address spaces. The size of the CSA area is established at system initialization time (IPL) and cannot change while the operating system is active. LSQA/SWA/subpool 228/subpool 230 This assortment of subpools, each with specific attributes, is used primarily by system functions when the functions require address space level storage isolation. Being below the line, these areas are addressable by programs running in 24-bit mode. User Region This area is obtainable by any program running in the user’s address space, including user key programs. It resides below the line and is therefore addressable by programs running in 24-bit mode. System Region This small area (usually only four pages) is reserved for use by the region control task of each address space. Prefixed Save Area (PSA) This area is often referred to as “Low Core.” The PSA is a common area of virtual storage from address zero through 8191 in every address space. There is one unique PSA for every processor installed in a system. The PSA maps architecturally fixed hardware and software storage locations for the processor. Because there is a unique PSA for each processor, from the view of a program running on z/OS, the contents of the PSA can change any time the program is dispatched on a different processor. This feature is unique to the PSA area and is accomplished through a unique DAT manipulation technique called prefixing. © Copyright IBM Corp., 2010. All rights reserved. Page 42 of 85
View full slide show




I/O Hardware  Incredible variety of I/O devices  Common concepts  Port  Bus (daisy chain or shared direct access)  Controller (host adapter)  I/O instructions control devices  Devices have addresses, used by  Direct I/O instructions  Memory-mapped I/O Operating System Concepts – 8th Edition 13.4 Silberschatz, Galvin and Gagne ©2009
View full slide show




Direct Memory Access • Direct Memory Access (DMA): External to the CPU Use idle bus cycles (cycle stealing) Act as a master on the bus Transfer blocks of data to or from memory without CPU intervention – Efficient for large data transfer, e.g. from disk  Cache usage allows the processor to leave enough memory bandwidth for DMA – – – – • How does DMA work?: – CPU sets up and supply device id, memory address, number of bytes – DMA controller (DMAC) starts the access and becomes bus master – For multiple byte transfer, the DMAC increment the address – DMAC interrupts the CPU upon completion 9 CPU sends a starting address, direction, and length count to DMAC. Then issues "start". CPU Memory DMAC IOC device DMAC provides handshake signals for Peripheral Controller, and Memory Addresses and handshake signals for Memory. For For multiple multiple bus bus system, system, each each bus bus controller controller often often contains contains DMA DMA control control logic logic
View full slide show




HDFS (Hadoop Distributed File System) is a distr file sys for commodity hdwr. Differences from other distr file sys are few but significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides hi thruput access to app data and is suitable for apps that have large data sets. HDFS relaxes a few POSIX requirements to enable streaming access to file system data. HDFS originally was infrastructure for Apache Nutch web search engine project, is part of Apache Hadoop Core http://hadoop.apache.org/core/ 2.1. Hardware Failure Hardware failure is the normal. An HDFS may consist of hundreds or thousands of server machines, each storing part of the file system’s data. There are many components and each component has a non-trivial prob of failure means that some component of HDFS is always non-functional. Detection of faults and quick, automatic recovery from them is core arch goal of HDFS. 2.2. Streaming Data Access Applications that run on HDFS need streaming access to their data sets. They are not general purpose applications that typically run on general purpose file systems. HDFS is designed more for batch processing rather than interactive use by users. The emphasis is on high throughput of data access rather than low latency of data access. POSIX imposes many hard requirements not needed for applications that are targeted for HDFS. POSIX semantics in a few key areas has been traded to increase data throughput rates. 2.3. Large Data Sets Apps on HDFS have large data sets, typically gigabytes to terabytes in size. Thus, HDFS is tuned to support large files. It provides high aggregate data bandwidth and scale to hundreds of nodes in a single cluster. It supports ~10 million files in a single instance. 2.4. Simple Coherency Model: HDFS apps need a write-once-read-many access model for files. A file once created, written, and closed need not be changed. This assumption simplifies data coherency issues and enables high throughput data access. A Map/Reduce application or a web crawler application fits perfectly with this model. There is a plan to support appending-writes to files in future [write once read many at file level] 2.5. “Moving Computation is Cheaper than Moving Data” A computation requested by an application is much more efficient if it is executed near the data it operates on. This is especially true when the size of the data set is huge. This minimizes network congestion and increases the overall throughput of the system. The assumption is that it is often better to migrate the computation closer to where the data is located rather than moving the data to where the app is running. HDFS provides interfaces for applications to move themselves closer to where the data is located. 2.6. Portability Across Heterogeneous Hardware and Software Platforms: HDFS has been designed to be easily portable from one platform to another. This facilitates widespread adoption of HDFS as a platform of choice for a large set of applications. 3. NameNode and DataNodes: HDFS has a master/slave architecture. An HDFS cluster consists of a single NameNode, a master server that manages the file system namespace and regulates access to files by clients. In addition, there are a number of DataNodes, usually one per node in the cluster, which manage storage attached to the nodes that they run on. HDFS exposes a file system namespace and allows user data to be stored in files. Internally, a file is 1 blocks stored in a set of DataNodes. The NameNode executes file system namespace operations like opening, closing, and renaming files and directories. It also determines the mapping of blocks to DataNodes. The DataNodes are responsible for serving read and write requests from the file system’s clients. The DataNodes also perform block creation, deletion, and replication upon instruction The NameNode and DataNode are pieces of software designed to run on commodity machines, typically run GNU/Linux operating system (OS). HDFS is built using the Java language; any machine that supports Java can run the NameNode or the DataNode software. Usage of the highly portable Java language means that HDFS can be deployed on a wide range of machines. A typical deployment has a dedicated machine that runs only the NameNode software. Each of the other machines in the cluster runs one instance of the DataNode software. The architecture does not preclude running multiple DataNodes on the same machine but in a real deployment that is rarely the case. The existence of a single NameNode in a cluster greatly simplifies the architecture of the system. The NameNode is the arbitrator and repository for all HDFS metadata. The system is designed in such a way that user data never flows through the NameNode. 4. The File System Namespace: HDFS supports a traditional hierarchical file organization. A user or an application can create directories and store files inside these directories. The file system namespace hierarchy is similar to most other existing file systems; one can create and remove files, move a file from one directory to another, or rename a file. HDFS does not yet implement user quotas or access permissions. HDFS does not support hard links or soft links. However, the HDFS architecture does not preclude implementing these features. The NameNode maintains the file system namespace. Any change to the file system namespace or its properties is recorded by the NameNode. An application can specify the number of replicas of a file that should be maintained by HDFS. The number of copies of a file is called the replication factor of that file. This info is stored by NameNode. 5. Data Replication: HDFS is designed to reliably store very large files across machines in a large cluster. It stores each file as a sequence of blocks; all blocks in a file except the last block are the same size. The blocks of a file are replicated for fault tolerance. The block size and replication factor are configurable per file. An application can specify the number of replicas of a file. The replication factor can be specified at file creation time and can be changed later. Files in HDFS are write-once and have strictly one writer at any time. The NameNode makes all decisions regarding replication of blocks. It periodically receives a Heartbeat and a Blockreport from each of the DataNodes in the cluster. Receipt of a Heartbeat implies that the DataNode is functioning properly. A Blockreport contains a list of all blocks on a DataNode
View full slide show




IEEE Reliability Test System 96-99 BUS 18 BUS 17 Total Load: 2,850 MW C BUS 21 BUS 23 A Plan Time Pe riod 0-72 h A Power G Energy Shed (MW) She d (MWh) 1,373 98,856 Total: 98,856 MWh 0-72 h 902 64,944 B 72-768 h 708 492,768 C 0-360 h BUS 16 230 kV A D B A A BUS 19 C B BUS 20 BUS 14 Synch. Cond. A BUS 13 BUS 15 E BUS 11 BUS 24 B Total: 557,712 MWh 756 BUS 22 272,160 BUS 12 B BUS 3 Total: 272,160 MW h C A BUS 9 BUS 10 BUS 6 cable BUS 4 F MW BUS 5 138 kV A B Attac +72h k C B +360h BUS 8 A cable BUS 7 +768h t BUS 1 BUS 2 Salmeron, Wood and Baldick (2004), IEEE 11 Transactions on Power Systems
View full slide show




I/O Hardware (Cont.)  I/O instructions control devices  Devices usually have registers where device driver places commands, addresses, and data to write, or read data from registers after command execution   Data-in register, data-out register, status register, control register  Typically 1-4 bytes, or FIFO buffer Devices have addresses, used by  Direct I/O instructions  Memory-mapped I/O  Device data and command registers mapped to processor address space  Especially for large address spaces (graphics) Operating System Concepts – 9th Edition 13.7 Silberschatz, Galvin and Gagne ©2013
View full slide show




A Two-Bus System Processor Memory Bus Processor Memory Bus Adaptor I/O Bus Bus Adaptor Bus Adaptor I/O Bus I/O Bus ° I/O buses tap into the processor-memory bus via bus adaptors: • Processor-memory bus: mainly for processor-memory traffic • I/O buses: provide expansion slots for I/O devices ° Apple Macintosh-II • NuBus: Processor, memory, and a few selected I/O devices • SCCI Bus: the rest of the I/O devices cs 152 buses.14 ©DAP & SIK 1995
View full slide show




A Computer with two Bus System Processor Memory Bus Main memory Proces s or Bus Adaptor I/O Bus Bus Adaptor I/O Bus Bus Adaptor I/O Bus • I/O buses tap into the processor - memory bus via bus adaptors: – Processor - memory bus: mainly for processor - memory traffic – I/O buses: provide expansion slots for I/O devices • Example: Apple Macintosh II – NuBus: Processor, memory, and a few selected I/O devices – SCCI Bus: the rest of the I/O devices CSE 45432 SUNY New Paltz 15
View full slide show




Ports and Connectors  What are USB ports? USB (universal serial bus) port can connect up to 127 different peripherals together with a single connector type PCs PCs typically typically have have six six to to eight USB USB ports on front front or or back back of of the the system system unit unit USB 2.0 p. 208 Single Single USB USB port port can can be be used used to attach multiple peripherals in in a daisy daisy chain chain First First USB USB device device connects connects to to USB USB port port on on computer computer Third Third USB USB device device connects connects to to second second USB USB device, device, and and so so on on Second Second USB USB device device connects connects to to first first USB USB device device Next
View full slide show




DBs, DWs are merging as In-memory DBs: SAP® In-Memory Computing Enabling Real-Time Computing SAP® In-Memory enables real-time computing by bringing together online transaction proc. OLTP (DB) and online analytical proc. OLAP (DW). Combining advances in hardware technology with SAP InMemory Computing empowers business – from shop floor to boardroom – by giving real-time bus. proc. instantaneous access to data-eliminating today’s info lag for your business. In-memory computing is already under way. The question isn’t if this revolution will impact businesses but when/ how. In-memory computing won’t be introduced because a co. can afford the technology. It will be because a business cannot afford to allow its competitors to adopt the it first. Here is sample of what in-memory computing can do for you: • Enable mixed workloads of analytics, operations, and performance management in a single software landscape. • Support smarter business decisions by providing increased visibility of very large volumes of business information • Enable users to react to business events more quickly through real-time analysis and reporting of operational data. • Deliver innovative real-time analysis and reporting. • Streamline IT landscape and reduce total cost of ownership. Product managers will still look at inventory and point-of-sale data, but in the future they will also receive,eg., tell customers broadcast dissatisfaction with a product over Twitter. Or they might be alerted to a negative product review released online that highlights some unpleasant product features requiring immediate action. From the other side, small businesses running real-time inventory reports will be able to announce to their Facebook and Twitter communities that a high demand product is available, how to order, and where to pick up. Bad movies have been able to enjoy a great opening weekend before crashing 2nd weekend when negative word-of-mouth feedback cools enthusiasm. That week-long grace period is about to disappear for silver screen flops. Consumer feedback won’t take a week, a day, or an hour. The very second showing of a movie could suffer from a noticeable falloff in attendance due to consumer criticism piped instantaneously through the new technologies. It will no longer be good enough to have weekend numbers ready for executives on Monday morning. Executives will run their own reports on revenue, Twitter their reviews, and by Monday morning have acted on their decisions. The final example is from the utilities industry: The most expensive energy a utilities provides is energy to meet unexpected demand during peak periods of consumption. If the company could analyze trends in power consumption based on real-time meter reads, it could offer – in real time – extra low rates for the week or month if they reduce their consumption during the following few hours. In manufacturing enterprises, in-memory computing tech will connect the shop floor to the boardroom, and the shop floor associate will have instant access to the same data as the board [[shop floor = daily transaction processing. Boardroom = executive data mining]]. The shop floor will then see the results of their actions reflected immediately in the relevant Key Performance Indicators (KPI). This advantage will become much more dramatic when we switch to electric cars; predictably, those cars are recharged the minute the owners return home from work. Hardware: blade servers and multicore CPUs and memory capacities measured in terabytes. Software: in-memory database with highly compressible row / column storage designed to maximize in-memory comp. tech. SAP BusinessObjects Event Insight software is key. In what used to be called exception reporting, the software deals with huge amounts of realtime data to determine immediate and appropriate action for a real-time situation. [[Both row and column storage! They convert to column-wise storage only for Long-Lived-High-Value data?]] Parallel processing takes place in the database layer rather than in the app layer - as it does in the client-server arch. Total cost is 30% lower than traditional RDBMSs due to: • Leaner hardware, less system capacity req., as mixed workloads of analytics, operations, performance mgmt is in a single system, which also reduces redundant data storage. [[Back to a single DB rather than a DB for TP and a DW for boardroom dec. sup.]] • Less extract transform load (ETL) between systems and fewer prebuilt reports, reducing support required to run sofwr. Report runtime improvements of up to 1000 times. Compression rates of up to a 10 times. Performance improvements expected even higher in SAP apps natively developed for inmemory DBs. Initial results: a reduction of computing time from hours to seconds. However, in-memory computing will not eliminate the need for data warehousing. Real-time reporting will solve old challenges and create new opportunities, but new challenges will arise. SAP HANA 1.0 software supports realtime database access to data from the SAP apps that support OLTP. Formerly, operational reporting functionality was transferred from OLTP applications to a data warehouse. With in-memory computing technology, this functionality is integrated back into the transaction system. Adopting in-memory computing results in an uncluttered arch based on a few, tightly aligned core systems enabled by service-oriented architecture (SOA) to provide harmonized, valid metadata and master data across business processes. Some of the most salient shifts and trends in future enterprise architectures will be: • A shift to BI self-service apps like data exploration, instead of static report solutions. • Central metadata and masterdata repositories that define the data architecture, allowing data stewards to work across all business units and all platforms Real-time in-memory computing technology will cause a decline Structured Query Language (SQL) satellite databases. The purpose of those databases as flexible, ad hoc, more business-oriented, less IT-static tools might still be required, but their offline status will be a disadvantage and will delay data updates. Some might argue that satellite systems with in-memory computing technology will take over from satellite SQL DBs. SAP Business Explorer tools that use in-memory computing technology represent a paradigm shift. Instead of waiting for IT to work on a long queue of support tickets to create new reports, business users can explore large data sets and define reports on the fly.
View full slide show




A Three-Bus System Processor Memory Bus Processor Memory Bus Adaptor Backplane Bus Bus Adaptor Bus Adaptor I/O Bus I/O Bus ° A small number of backplane buses tap into the processor-memory bus • Processor-memory bus is used for processor memory traffic • I/O buses are connected to the backplane bus ° Advantage: loading on the processor bus is greatly reduced cs 152 buses.15 ©DAP & SIK 1995
View full slide show