Disk Structure  Disk drives are addressed as large 1-dimensional arrays of logical blocks, where the logical block is the smallest unit of transfer   Low-level formatting creates logical blocks on physical media The 1-dimensional array of logical blocks is mapped into the sectors of the disk sequentially  Sector 0 is the first sector of the first track on the outermost cylinder  Mapping proceeds in order through that track, then the rest of the tracks in that cylinder, and then through the rest of the cylinders from outermost to innermost  Logical to physical address should be easy  Except for bad sectors  Non-constant # of sectors per track via constant angular velocity Operating System Concepts Essentials – 2nd Edition 9.11 Silberschatz, Galvin and Gagne ©2013
View full slide show




Disk Structure  Disk drives are addressed as large 1-dimensional arrays of logical blocks, where the logical block is the smallest unit of transfer  The 1-dimensional array of logical blocks is mapped into the sectors of the disk sequentially  Sector 0 is the first sector of the first track on the outermost cylinder  Mapping proceeds in order through that track, then the rest of the tracks in that cylinder, and then through the rest of the cylinders from outermost to innermost  Logical to physical address should be easy  Except for bad sectors  Non-constant # of sectors per track via constant angular velocity Operating System Concepts Essentials – 8 th Edition 11.10 Silberschatz, Galvin and Gagne ©2011
View full slide show




Disk Structure  Disk drives are addressed as large 1-dimensional arrays of logical blocks, where the logical block is the smallest unit of transfer  The 1-dimensional array of logical blocks is mapped into the sectors of the disk sequentially  Sector 0 is the first sector of the first track on the outermost cylinder  Mapping proceeds in order through that track, then the rest of the tracks in that cylinder, and then through the rest of the cylinders from outermost to innermost Operating System Concepts with Java – 8th Edition 12.7 Silberschatz, Galvin and Gagne ©2009
View full slide show




Time Barbie Tootle Hayes Cape Cartoon Room I Cartoon Room II Suzanne M. Scharer Rosa M. Ailabouni Monday Aug 5th, 2013 10:10-11:50 A1L-A Analog Circuits I Chr: Ming Gu, Shantanu Chakrabartty Track: Analog and Mixed Signal Integrated Circuits A1L-B Low Power Digital Circuit Design Techniques A1L-C Chr: Joanne Degroat Student Contest I Track: Digital Integrated Chr: Mohammed Ismail Circuits, SoC and NoC Track: INVITED ONLY A1L-D Design and Analysis for Power Systems and Power Electronics Chr: Hoi Lee, Ayman Fayed Track: Power Systems and Power Electronics A1L-E Design and Analysis of Linear and Non-Linear Systems Chr: Samuel Palermo Track: Linear and Non-linear Circuits and Systems A1L-F Emerging Technologues Chr: Khaled Salama Track: Emerging Technologies Monday Aug 5th, 2013 13:10-14:50 A2L-A Analog Circuits II Chr: Ming Gu, Shantanu Chakrabartty Track: Analog and Mixed Signal Integrated Circuits A2L-B Low Power VLSI Design Methodology Chr: Genevieve Sapijaszko Track: Digital Integrated Circuits, SoC and NoC A2L-C Student Contest II Chr: Sleiman Bou-Sleiman Track: INVITED ONLY A2L-D Power Management and Energy Harvesting Chr: Ayman Fayed, Hoi Lee Track: Power Management and Energy Harvesting A2L-E Oscillators and Chaotic Systems Chr: Samuel Palermo, Warsame Ali Track: Linear and Non-linear Circuits and Systems A2L-F Bioengineering Systems Chr: Khaled Salama Track: Bioengineering Systems and Bio Chips A4L-A Analog Design Techniques I Chr: Dong Ha Track: Analog and Mixed Signal Integrated Circuits A4L-B Imaging and Wireless Sensors Chr: Igor Filanovsky Track: Analog and Mixed Signal Integrated Circuits A4L-C Special Session: Characterization of Nano Materials and Circuits Chr: Nayla El-Kork Track: SPECIAL SESSION A4L-D Special Session: Power Management and Energy Harvesting Chr: Paul Furth Track: SPECIAL SESSION A4L-E Communication and Signal Processing Circuits Chr: Samuel Palermo Track: Linear and Non-linear Circuits and Systems A4L-F Sensing and Measurement of Biological Signals Chr: Hoda Abdel-Aty-Zohdy Track: Bioengineering Systems and Bio Chips B2L-A Analog Design Techniques II Chr: Valencia Koomson Track: Analog and Mixed Signal Integrated Circuits B2L-B VLSI Design Reliability Chr: Shantanu Chakrabartty, Gursharan Reehal Track: Digital Integrated Circuits, SoC and NoC B2L-D B2L-C Special Session: University and Delta-Sigma Modulators Industry Training in the Art of Chr: Vishal Saxena Electronics Track: Analog and Mixed Signal Chr: Steven Bibyk Integrated Circuits Track: SPECIAL SESSION B2L-E Radio Frequency Integrated Circuits Chr: Nathan Neihart, Mona Hella Track: RFICs, Microwave, and Optical Systems B2L-F Bio-inspired Green Technologies Chr: Hoda Abdel-Aty-Zohdy Track: Bio-inspired Green Technologies B3L-A Analog Design Techniques III Chr: Valencia Koomson Track: Analog and Mixed Signal Integrated Circuits B3L-B VLSI Design, Routing, and Testing Chr: Nader Rafla Track: Programmable Logic, VLSI, CAD and Layout B3L-C Special Session: High-Precision and High-Speed Data Converters I Chr: Samuel Palermo Track: SPECIAL SESSION B3L-D B3L-E Special Session: Advancing the RF/Optical Devices and Circuits Frontiers of Solar Energy Chr: Mona Hella, Nathan Neihart Chr: Michael Soderstrand Track: RFICs, Microwave, and Track: SPECIAL SESSION Optical Systems B5L-A Nyquist-Rate Data Converters Chr: Vishal Saxena Track: Analog and Mixed Signal Integrated Circuits B5L-B Digital Circuits Chr: Nader Rafla Track: Programmable Logic, VLSI, CAD and Layout B5L-C Special Session: High-Precision and High-Speed Data Converters II Chr: Samuel Palermo Track: SPECIAL SESSION B5L-D Special Session: RF-FPGA Circuits and Systems for Enhancing Access to Radio Spectrum (CAS-EARS) Chr: Arjuna Madanayake, Vijay Devabhaktuni Track: SPECIAL SESSION B5L-E B5L-F Analog and RF Circuit Memristors, DG-MOSFETS and Techniques Graphine FETs Chr: Igor Filanovsky Chr: Reyad El-Khazali Track: Analog and Mixed Signal Track: Nanoelectronics and Integrated Circuits Nanotechnology C2L-A Phase Locked Loops Chr: Chung-Chih Hung Track: Analog and Mixed Signal Integrated Circuits C2L-B Computer Arithmetic and Cryptography Chr: George Purdy Track: Programmable Logic, VLSI, CAD and Layout C2L-C Special Session: Reversible Computing Chr: Himanshu Thapliyal Track: SPECIAL SESSION C2L-D Special Session: Self-healing and Self-Adaptive Circuits and Systems Chr: Abhilash Goyal, Abhijit Chatterjee Track: SPECIAL SESSION C2L-E Digital Signal Processing-Media and Control Chr: Wasfy Mikhael, Steven Bibyk Track: Digital Signal Processing C2L-F Advances in Communications and Wireless Systems Chr: Sami Muhaidat Track: Communication and Wireless Systems C3L-A SAR Analog-to-Digital Converters Chr: Vishal Saxena Track: Analog and Mixed Signal Integrated Circuits C3L-B Real Time Systems Chr: Brian Dupaix, Abhilash Goyal Track: System Architectures C3L-C Image Processing and Interpretation Chr: Annajirao Garimella Track: Image Processing and Multimedia Systems C3L-D Special Session: Verification and Trusted Mixed Signal Electronics Development Chr: Greg Creech, Steven Bibyk Track: SPECIAL SESSION C3L-E Digital Signal Processing I Chr: Ying Liu Track: Digital Signal Processing C3L-F Wireless Systems I Chr: Sami Muhaidat Track: Communication and Wireless Systems C5L-A Wireless Systems II Chr: Sami Muhaidat Track: Communication and Wireless Systems C5L-B System Architectures Chr: Swarup Bhunia, Abhilash Goyal Track: System Architectures C5L-C Image Embedding Compression and Analysis Chr: Annajirao Garimella Track: Image Processing and Multimedia Systems C5L-D Low Power Datapath Design Chr: Wasfy Mikhael Track: Digital Integrated Circuits, SoC and NoC C5L-E Digital Signal Processing II Chr: Moataz AbdelWahab Track: Digital Signal Processing C5L-F Advances in Control Systems, Mechatronics, and Robotics Chr: Charna Parkey, Genevieve Sapijaszko Track: Control Systems, Mechatronics, and Robotics Monday Aug 5th, 2013 16:00-17:40 Tuesday Aug 6th, 2013 10:10-11:50 Tuesday Aug 6th, 2013 13:10-14:50 Tuesday Aug 6th, 2013 16:00-17:40 Wednesday Aug 7th, 2013 10:10-11:50 Wednesday Aug 7th, 2013 13:10-14:50 Wednesday Aug 7th, 2013 16:00-17:40 B3L-F Carbon Nanotube-based Sensors and Beyond Chr: Nayla El-Kork Track: Nanoelectronics and Nanotechnology 5
View full slide show




Disk Structure • • • • • Cylinder: the set of tracks that all the heads are currently located at. Track: A ring on a disk where data can be written Sector: The smallest transfer unit of data accessed in a block Cluster: A group of sectors the operating system treats as a unit Organization Choices – Sector mapping (One dimension array of logical blocks) • 0 is first sector, track 0 of the outermost cylinder. • Subsequent sectors map through tracks, through cylinders, in an outer to inner direction. – Sector counts and density • fixed sectors per track: varying densities • Varied sectors per track: outer tracks have more sectors; constant density – Bad block management • Sector sparing: replace bad sectors with spares in the same cylinder • Sector slipping: copy all sectors down to the next spare
View full slide show




RAPTOR Syntax and Semantics - Arrays Array variable - Array variables are used to store many values (of the same type) without having to have many variable names. Instead of many variables names a count-controlled loop is used to gain access (index) the individual elements (values) of an array variable. RAPTOR has one and two dimensional arrays of numbers. A one dimensional array can be thought of as a sequence (or a list). A two dimensional array can be thought of as a table (grid or matrix). To create an array variable in RAPTOR, use it like an array variable. i.e. have an index, ex. Score[1], Values[x], Matrix[3,4], etc. All array variables are indexed starting with 1 and go up to the largest index used so far. RAPTOR array variables grow in size as needed. The assignment statement GPAs[24] ← 4.0 assigns the value 4.0 to the 24th element of the array GPAs. If the array variable GPAs had not been used before then the other 23 elements of the GPAs array are initialized to 0 at the same time. i.e. The array variable GPAs would have the following values: 1 2 3 4… Array variables in action- Arrays and count-controlled loop statements were made for each other. Notice in each example below the connection between the Loop Control Variable and the array index! Notice how the Length_Of function can be used in the count-controlled loop test! Notice that each example below is a count-controlled loop and has an Initialize, Test, Execute, and Modify part (I.T.E.M)! Assigning values to an array variable Reading values into an array variable Writing out an array variable’s values Computing the total and average of an array variable’s values Index ← 1 Index ← 1 Index ← 1 Total ← 0 Loop Loop Loop Index ← 1 PUT “The value of the array at position “ + Index + “ is “ + GPAs[Index] Loop GPAs[Index] ← 4.0 “Enter the GPA of student “” + Index + “: “ GET GPAs[Index] Index >= 24 Index >= 24 Index >= Length_Of (GPAs) Index ← Index + 1 Index ← Index + 1 Index ← Index + 1 Total ← Total + GPAs[Index] Index >= Length_Of(GPAs) Index ← Index + 1 … 23 24 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4.0 The initialization of previous elements to 0 happens only when the array variable is created. Successive assignment statements to the GPAs variable affect only the individual element listed. For example, the following successive assignment statements GPAs[20] GPAs[11] ← ← 1.7 3.2 would place the value 1.7 into the 20th position of the array, and would place the value 3.2 into the 11th position of the array. i.e. GPAs[20] ← 1.7 GPAs[11] ← 3.2 1 2 3 4… … 23 24 Initialize the elements of a two dimensional array (A two dimensional array requires two loops) Row ← 1 Loop Average ← Total / Length_Of(GPAs) Find the largest value of all the values in an array variable Find the INDEX of the largest value of all the values in an array variable Highest_GPA ← GPAs[1] Highest_GPA_Index ←1 Index ← 1 Index ← 1 Loop Loop GPAs[Index] > Highest_GPA GPAs[Index] >= GPAs[Highest_GPA_Index] Column ← 1 Loop 0 0 0 0 0 0 0 0 0 0 3.2 0 0 0 0 0 0 0 0 1.7 0 0 0 4.0 An array variable name, like GPAs, refers to ALL elements of the array. Adding an index (position) to the array variable enables you to refer to any specific element of the array variable. Two dimensional arrays work similarly. i.e. Table[7,2] refers to the element in the 7 th row and 2nd column. Individual elements of an array can be used exactly like any other variable. E.g. the array element GPAs[5] can be used anywhere the number variable X can be used. The Length_Of function can be used to determine (and return) the number of elements that are associated with a particular array variable. For example, after all the above, Length_Of(GPAs) is 24. Matrix[Row, Column] ← 1 Column >= 20 Column ← Column + 1 Highest_GPA ← GPAs[Index] Highest_GPA_Index ← Index Index >= Length_Of(GPAs) Index >= Length_Of(GPAs) Index ← Index + 1 Index ← Index + 1 PUT “The highest GPA is “ + Highest_GPA¶ PUT “The highest GPA is “ + GPAs[Highest_GPA_Index] + “ it is at position “ + Highest_GPA_Index¶ Row >= 20 Row ← Row + 1
View full slide show




Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks on Blocks! Dynamically Rearranging Synteny Blocks in Comparative Genomes Nick Egan’s Final Project Presentation for BIO 131 Intro to Computational Biology Taught by Anna Ritz
View full slide show




Preamble “Post-amble” Block Execution: 3 Detail Observing Block Observing Block “Post-amble” “Post-amble” 3 Observing Block Observing Block ok Measurement Set ready “Post-amble” EVLA Data Processing PDR Observing Observing Block Block Observing Observing Block Block Failed! Preamble “Post-amble” Preamble ok ?4 5 Preamble ready Preamble Observing Observing Block Block Observing Observing Block Block Observing Block Observing Block Measurement Set “Post-amble” “Post-amble” Preamble Preamble “Post-amble” Measurement Set “Post-amble” “Post-amble” “Post-amble” July 18 - 19, 2002 2 2 Observing Observing Block Block Block Observing Observing Observing Block Block ok Archive: Preamble Observing Block Observing Block 34 ready Preamble “Post-amble” 1 3 Observing Block Observing Observing Block Block Observing Block Observing Observing Block Block ready Preamble Execution: Preamble ready Observing Observing Block Block Observing Observing Block Block Preamble Observing Block Observing Block 22 “Post-amble” “Post-amble” Preamble Preamble 1 “Post-amble” Preamble Input Queue: ok Measurement Set Boyd Waters 13
View full slide show




Disk Management     Low-level formatting, or physical formatting — Dividing a disk into sectors that the disk controller can read and write  Each sector can hold header information, plus data, plus error correction code (ECC)  Usually 512 bytes of data but can be selectable To use a disk to hold files, the operating system still needs to record its own data structures on the disk  Partition the disk into one or more groups of cylinders, each treated as a logical disk  Logical formatting or “making a file system”  To increase efficiency most file systems group blocks into clusters  Disk I/O done in blocks  File I/O done in clusters Boot block initializes system  The bootstrap is stored in ROM  Bootstrap loader program stored in boot blocks of boot partition Methods such as sector sparing used to handle bad blocks Operating System Concepts Essentials – 8 th Edition 11.28 Silberschatz, Galvin and Gagne ©2011
View full slide show




EmptyPersistentList public public class class NonEmptyPersistentList NonEmptyPersistentList extends extends AbstractPersistentList AbstractPersistentList { { private private final final E E first; first; private private final final PersistentList PersistentList rest; rest; public public NonEmptyPersistentList(E NonEmptyPersistentList(E first, first, PersistentList PersistentList rest) rest) { { this.first this.first = = first; first; this.rest this.rest = = rest; rest; } } public public E E get(int get(int index) index) { { if(index if(index == == 0) 0) { { return return first; first; } } else else { { return return rest.get(index-1); rest.get(index-1); } } } } public public int int size() size() { { return return 1 1 + + rest.size(); rest.size(); } } public public E E first() first() { { return return first; first; } } public public PersistentList PersistentList rest() rest() { { return return rest; rest; } } public public PersistentList PersistentList append(PersistentList append(PersistentList suffix) suffix) { { return new NonEmptyPersistentList(first, rest.append(suffix)); return new NonEmptyPersistentList(first, rest.append(suffix)); } } public public PersistentList PersistentList reverse() reverse() { { return return rest.reverse().append(new rest.reverse().append(new NonEmptyPersistentList(first, NonEmptyPersistentList(first, new new EmptyPersistentList())); EmptyPersistentList())); } } public public boolean boolean contains(Object contains(Object obj) obj) { { return return first.equals(obj) first.equals(obj) || || rest.contains(obj); rest.contains(obj); } } public public boolean boolean isEmpty() isEmpty() { { return return false; false; } } public public PersistentList PersistentList add(E add(E elem) elem) { { return return new new NonEmptyPersistentList(elem, NonEmptyPersistentList(elem, this); this); } } } }
View full slide show




Array Lesson 1 Outline 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. Array Lesson 1 Outline Mean of a List of Numbers Mean: Declarations Mean: Greeting, Input Mean: Calculation Mean: Output Mean: Compile, Run Mean: 5 Input Values Mean: 7 Input Values Mean: One Line Different Mean: Compile, Run for 5 Mean: Compile, Run for 7 Scalars #1 Scalars #2 Another Scalar Example A Similar Program, with Multiplication A Similar Program, with a Twist Arrays Array Element Properties 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. Array Properties #1 Array Properties #2 Array Properties #3 Array Properties #4 Array Properties #5 Array Indices #1 Array Indices #2 Multidimensional Arrays & 1D Arrays Array Declarations #1 Array Declarations #2 Array Declarations #3 Assigning a Value to an Array Element Array Element Assignment Example Getting Array Element Value with scanf Array Element scanf Example #1 Array Element scanf Example #2 for Loops for Tasks on Arrays #1 for Loops for Tasks on Arrays #2 Another for/Array Example #1 Another for/Array Example #2 Another for/Array Example #3 Don’t Need to Use Entire Declared Length Array Lesson 1 CS1313 Spring 2019 1
View full slide show




Disk Management  Low-level formatting, or physical formatting — Dividing a disk into sectors that the disk controller can read and write  To use a disk to hold files, the operating system still needs to record its own data structures on the disk  Partition the disk into one or more groups of cylinders  Logical formatting or “making a file system”  To increase efficiency most file systems group blocks into clusters  Disk I/O done in blocks  File I/O done in clusters  Boot block initializes system  The bootstrap is stored in ROM  Bootstrap loader program  Methods such as sector sparing used to handle bad blocks Operating System Concepts with Java – 8th Edition 12.18 Silberschatz, Galvin and Gagne ©2009
View full slide show




Transparent Scalability  Hardware is free to assign blocks to any SM (processor)  A kernel scales across any number of parallel processors Device Kernel grid Device Block 0 Block 1 Block 2 Block 3 Block 0 Block 1 Block 4 Block 5 Block 6 Block 7 Block 2 Block 3 Block 4 Block 5 Block 6 Block 7 26 time Block 0 Block 1 Block 2 Block 3 Block 4 Block 5 Block 6 Block 7 Each block can execute in any order relative to other blocks.
View full slide show




Transparent Scalability Hardware is free to assign blocks to any processor at any time  A kernel scales across any number of parallel processors Device Device Kernel grid Block 0 Block 1 Block 2 Block 3 Block 0 Block 2 Block 1 Block 3 Block 4 Block 5 Block 6 Block 7  Block 4 Block 5 Block 6 Block 7 time Block 0 Block 1 Block 2 Block 3 Block 4 Block 5 Block 6 Block 7 Each block can execute in any CUDA Tools and Threads – Slide order relative 69
View full slide show




I/O Example: Disk Drives Typically: • 1000 - 5000 tracks per surface Platters •64 - 200 sectors per track •512 Bytes per sector Tracks Platter Sectors Track • To access data: – wait time: until disk is not used for other transactions – seek: position head over the proper track – rotational latency: wait for desired sector – transfer: grab the data (one or more sectors) – controller time CSE 45432 SUNY New Paltz •All tracks have the same number of sectors •real/write heads •Cylinders 7
View full slide show




Disk Management   Low-level formatting, or physical formatting — Dividing a disk into sectors that the disk controller can read and write  Each sector can hold header information, plus data, plus error correction code (ECC)  Usually 512 bytes of data but can be selectable To use a disk to hold files, the operating system still needs to record its own data structures on the disk  Partition the disk into one or more groups of cylinders, each treated as a logical disk  Logical formatting or “making a file system”  To increase efficiency most file systems group blocks into clusters  Disk I/O done in blocks  File I/O done in clusters Operating System Concepts Essentials – 2nd Edition 9.29 Silberschatz, Galvin and Gagne ©2013
View full slide show




HDFS (Hadoop Distributed File System) is a distr file sys for commodity hdwr. Differences from other distr file sys are few but significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides hi thruput access to app data and is suitable for apps that have large data sets. HDFS relaxes a few POSIX requirements to enable streaming access to file system data. HDFS originally was infrastructure for Apache Nutch web search engine project, is part of Apache Hadoop Core http://hadoop.apache.org/core/ 2.1. Hardware Failure Hardware failure is the normal. An HDFS may consist of hundreds or thousands of server machines, each storing part of the file system’s data. There are many components and each component has a non-trivial prob of failure means that some component of HDFS is always non-functional. Detection of faults and quick, automatic recovery from them is core arch goal of HDFS. 2.2. Streaming Data Access Applications that run on HDFS need streaming access to their data sets. They are not general purpose applications that typically run on general purpose file systems. HDFS is designed more for batch processing rather than interactive use by users. The emphasis is on high throughput of data access rather than low latency of data access. POSIX imposes many hard requirements not needed for applications that are targeted for HDFS. POSIX semantics in a few key areas has been traded to increase data throughput rates. 2.3. Large Data Sets Apps on HDFS have large data sets, typically gigabytes to terabytes in size. Thus, HDFS is tuned to support large files. It provides high aggregate data bandwidth and scale to hundreds of nodes in a single cluster. It supports ~10 million files in a single instance. 2.4. Simple Coherency Model: HDFS apps need a write-once-read-many access model for files. A file once created, written, and closed need not be changed. This assumption simplifies data coherency issues and enables high throughput data access. A Map/Reduce application or a web crawler application fits perfectly with this model. There is a plan to support appending-writes to files in future [write once read many at file level] 2.5. “Moving Computation is Cheaper than Moving Data” A computation requested by an application is much more efficient if it is executed near the data it operates on. This is especially true when the size of the data set is huge. This minimizes network congestion and increases the overall throughput of the system. The assumption is that it is often better to migrate the computation closer to where the data is located rather than moving the data to where the app is running. HDFS provides interfaces for applications to move themselves closer to where the data is located. 2.6. Portability Across Heterogeneous Hardware and Software Platforms: HDFS has been designed to be easily portable from one platform to another. This facilitates widespread adoption of HDFS as a platform of choice for a large set of applications. 3. NameNode and DataNodes: HDFS has a master/slave architecture. An HDFS cluster consists of a single NameNode, a master server that manages the file system namespace and regulates access to files by clients. In addition, there are a number of DataNodes, usually one per node in the cluster, which manage storage attached to the nodes that they run on. HDFS exposes a file system namespace and allows user data to be stored in files. Internally, a file is 1 blocks stored in a set of DataNodes. The NameNode executes file system namespace operations like opening, closing, and renaming files and directories. It also determines the mapping of blocks to DataNodes. The DataNodes are responsible for serving read and write requests from the file system’s clients. The DataNodes also perform block creation, deletion, and replication upon instruction The NameNode and DataNode are pieces of software designed to run on commodity machines, typically run GNU/Linux operating system (OS). HDFS is built using the Java language; any machine that supports Java can run the NameNode or the DataNode software. Usage of the highly portable Java language means that HDFS can be deployed on a wide range of machines. A typical deployment has a dedicated machine that runs only the NameNode software. Each of the other machines in the cluster runs one instance of the DataNode software. The architecture does not preclude running multiple DataNodes on the same machine but in a real deployment that is rarely the case. The existence of a single NameNode in a cluster greatly simplifies the architecture of the system. The NameNode is the arbitrator and repository for all HDFS metadata. The system is designed in such a way that user data never flows through the NameNode. 4. The File System Namespace: HDFS supports a traditional hierarchical file organization. A user or an application can create directories and store files inside these directories. The file system namespace hierarchy is similar to most other existing file systems; one can create and remove files, move a file from one directory to another, or rename a file. HDFS does not yet implement user quotas or access permissions. HDFS does not support hard links or soft links. However, the HDFS architecture does not preclude implementing these features. The NameNode maintains the file system namespace. Any change to the file system namespace or its properties is recorded by the NameNode. An application can specify the number of replicas of a file that should be maintained by HDFS. The number of copies of a file is called the replication factor of that file. This info is stored by NameNode. 5. Data Replication: HDFS is designed to reliably store very large files across machines in a large cluster. It stores each file as a sequence of blocks; all blocks in a file except the last block are the same size. The blocks of a file are replicated for fault tolerance. The block size and replication factor are configurable per file. An application can specify the number of replicas of a file. The replication factor can be specified at file creation time and can be changed later. Files in HDFS are write-once and have strictly one writer at any time. The NameNode makes all decisions regarding replication of blocks. It periodically receives a Heartbeat and a Blockreport from each of the DataNodes in the cluster. Receipt of a Heartbeat implies that the DataNode is functioning properly. A Blockreport contains a list of all blocks on a DataNode
View full slide show




Z502 Disk Structure Header For Directory or File Index Sector – showing the layout of disk sectors for a single level index Byte Offset Bytes in Field Description 0 2 Holds the sector number for a block in the file, a directory header, or 0 2 2 Holds the sector number for a block in the file, a directory header, or 0 4 2 Holds the sector number for a block in the file, a directory header, or 0 6 2 Holds the sector number for a block in the file, a directory header, or 0 8 2 Holds the sector number for a block in the file, a directory header, or 0 10 2 Holds the sector number for a block in the file, a directory header, or 0 12 2 Holds the sector number for a block in the file, a directory header, or 0 14 2 Holds the sector number for a block in the file, a directory header, or 0 Locations not pointing to a sector must be set to 0. The two bytes for each sector number obey the rules on Slide 8. So a pointer to Sector 128. would look like 10000000 00000000. Byte 0 = 80x, Byte 1 = 0x. 14
View full slide show




C:\UMBC\331\java> java.ext.dirs=C:\JDK1.2\JRE\lib\ext java.io.tmpdir=C:\WINDOWS\TEMP\ os.name=Windows 95 java.vendor=Sun Microsystems Inc. java.awt.printerjob=sun.awt.windows.WPrinterJob java.library.path=C:\JDK1.2\BIN;.;C:\WINDOWS\SYSTEM;C:\... java.vm.specification.vendor=Sun Microsystems Inc. sun.io.unicode.encoding=UnicodeLittle file.encoding=Cp1252 java.specification.vendor=Sun Microsystems Inc. user.language=en user.name=nicholas java.vendor.url.bug=http://java.sun.com/cgi-bin/bugreport... java.vm.name=Classic VM java.class.version=46.0 java.vm.specification.name=Java Virtual Machine Specification sun.boot.library.path=C:\JDK1.2\JRE\bin os.version=4.10 java.vm.version=1.2 java.vm.info=build JDK-1.2-V, native threads, symcjit java.compiler=symcjit path.separator=; file.separator=\ user.dir=C:\UMBC\331\java sun.boot.class.path=C:\JDK1.2\JRE\lib\rt.jar;C:\JDK1.2\JR... user.name=nicholas user.home=C:\WINDOWS C:\UMBC\331\java>java envSnoop -- listing properties -java.specification.name=Java Platform API Specification awt.toolkit=sun.awt.windows.WToolkit java.version=1.2 java.awt.graphicsenv=sun.awt.Win32GraphicsEnvironment user.timezone=America/New_York java.specification.version=1.2 java.vm.vendor=Sun Microsystems Inc. user.home=C:\WINDOWS java.vm.specification.version=1.0 os.arch=x86 java.awt.fonts= java.vendor.url=http://java.sun.com/ user.region=US file.encoding.pkg=sun.io java.home=C:\JDK1.2\JRE java.class.path=C:\Program Files\PhotoDeluxe 2.0\Adob... line.separator=
View full slide show




The Linux ext3 File System (Cont.)  The main differences between ext2fs and FFS concern their disk allocation policies  In ffs, the disk is allocated to files in blocks of 8Kb, with blocks being subdivided into fragments of 1Kb to store small files or partially filled blocks at the end of a file  ext3 does not use fragments; it performs its allocations in smaller units  The default block size on ext3 varies as a function of total size of file system with support for 1, 2, 4 and 8 KB blocks  ext3 uses cluster allocation policies designed to place logically adjacent blocks of a file into physically adjacent blocks on disk, so that it can submit an I/O request for several disk blocks as a single operation on a block group  Maintains bit map of free blocks in a block group, searches for free byte to allocate at least 8 blocks at a time Operating System Concepts Essentials – 2nd Edition 15.49 Silberschatz, Galvin and Gagne ©2013
View full slide show