Source code of Windows XP (NT5)
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

2230 lines
49 KiB

  1. /*++
  2. Copyright (c) 1995 Microsoft Corporation
  3. Module Name:
  4. hashmap.h
  5. Abstract:
  6. This module contains class declarations/definitions for
  7. CHashMap
  8. CMsgArtMap
  9. CHistory
  10. **** Schedule ****
  11. Hash function Free (ripped out from INN code)
  12. Core Hash Engine: 2 weeks
  13. Initialization/Setup
  14. Collision resolution
  15. Double hashing/
  16. linear probing
  17. Insertion
  18. Page Splitting
  19. Search
  20. Deletion
  21. Directory setup
  22. Article Mapping 1 week
  23. Leaf Page Setup
  24. Leaf Page Compaction
  25. Recovery
  26. Statistics
  27. History 1 week
  28. Leaf Page Setup
  29. Insertion
  30. Crawler/Expiration/Deletion
  31. Statistics
  32. Unit Testing 2 weeks
  33. Get a large set of msg ids
  34. Write an app to
  35. try out hash fn and make sure distribution is acceptable
  36. test page splitting and directory growing operations
  37. test compaction code
  38. test multithreaded operations
  39. get raw speed of table lookup (lookup/sec)
  40. detect map file corruption
  41. rebuilding of map file
  42. test history crawler delete and compaction operations
  43. Misc design stuff
  44. At startup, figure out the system page size
  45. Regkey for size of mapping file
  46. Add collision info to statistics
  47. Backup history file
  48. **** Hash algorithm ****
  49. Overview:
  50. The hashing mechanism is made up of 2 structures: the directory,
  51. and the leaf pages. The directory is made up of 2**n entries,
  52. where n is the first n bits of a hash value. Each directory
  53. entry points to a leaf page. (A leaf page can be pointed to by
  54. more than one entry).
  55. A leaf page contains up to x entries. When the page is about to be
  56. full (determined by number of entries or by space remaining), the
  57. page is split into two. After the page is split, the directory pointers
  58. will also have to be modified to account for the new page.
  59. This scheme ensures that we take at most 2 page faults for each search.
  60. Implementation details:
  61. The leaf page can accomodate up to 128 entries. The last m=7 bits
  62. of the hash value is used to hash into the list. We will use
  63. linear probing with p=1 to resolve collisions.
  64. The leaf page has fields called HashPrefix and PageDepth. The HashPrefix
  65. contains the first PageDepth bits of a hash value which this page
  66. represents. PageDepth <= n. If PageDepth < n, then that means that
  67. more than one entry points to it.
  68. If PageDepth == n and we need to split the page, then this would
  69. necessitate rebuilding the directory structure so the condition
  70. PageDepth <= n will still be true after the split.
  71. We are using the first page of the mapping file as a reserved page.
  72. Here, we will store statistics information etc.
  73. **** Recovery strategy ****
  74. Detection:
  75. The first page of the map file will be reserved. This will contain
  76. timestamps and statistics. There will be an entry to indicate
  77. whether a clean shutdown took place. If during startup, this
  78. flag indicates otherwise, we need to do a possible rebuild.
  79. ** We can also think of putting something in the registry
  80. to indicate this. Maybe we'll do both. **
  81. There will be a LastWrite timestamp in this page. After each flush,
  82. the timestamp will be updated. This will give us an indication on
  83. when the last complete write took place.
  84. Rebuilding the mapping file:
  85. There are three levels of map file rebuilding. This can be
  86. specified by the operator.
  87. Level 1 rebuild
  88. Using the timestamp, find all the articles file time stamps
  89. which were created after this and add them to the lookup table.
  90. Level 2 rebuild
  91. A cursory rebuild assumes that all the XOver files in each
  92. newsgroup is valid. The rebuilding will involve going over
  93. each XOver file (about 10000 of them), and rebuild the table
  94. with xover entries.
  95. Level 3 rebuild
  96. The more extensive rebuild involves going through each and every
  97. article in every newsgroup. This will probably involve at least
  98. a million files and would substantially take longer to do.
  99. How do we know if we need to upgrade to a higher level?
  100. Well, the hash table will know how many article entries it
  101. has and there should be some way of figuring out how many
  102. articles are stored in the disk. If they don't match, then
  103. it's time to upgrade.
  104. **** Registry Keys ****
  105. MappingFileSize DWORD Size of mapping files
  106. HashFileName REG_SZ Name of hash files
  107. InitialPageDepth DWORD Initial depth of leaf nodes
  108. Did they existed ?
  109. Author:
  110. Johnson Apacible (JohnsonA) 07-Sept-1995
  111. Revision History:
  112. Alex Wetmore (awetmore) 27-Aug-1996
  113. - seperated from nntp server
  114. - removed dependencies for this include file on others...
  115. --*/
  116. #ifndef _HASHMAP_
  117. #define _HASHMAP_
  118. #include "smartptr.h"
  119. #ifdef _USE_RWNH_
  120. #include "rwnew.h"
  121. typedef class CShareLockNH _RWLOCK ;
  122. #else
  123. #include "rw.h"
  124. typedef class CShareLock _RWLOCK ;
  125. #endif
  126. //
  127. // type declarations
  128. //
  129. typedef DWORD HASH_VALUE;
  130. //
  131. // manifest constants for hash stuff
  132. //
  133. #define KB 1024
  134. #define HASH_PAGE_SIZE (4 * KB)
  135. #define NUM_PAGES_PER_IO 64
  136. #define MAX_MSGID_LEN 255
  137. #define MIN_HASH_FILE_SIZE ((1+(1<<NUM_TOP_DIR_BITS)) * HASH_PAGE_SIZE)
  138. #define LEAF_ENTRY_BITS 7
  139. #define MAX_LEAF_ENTRIES (1 << LEAF_ENTRY_BITS)
  140. #define LEAF_ENTRY_MASK (MAX_LEAF_ENTRIES - 1)
  141. #define MAX_XPOST_GROUPS 255
  142. //
  143. // Num of entries, bytes, and size of fragments to trigger page split.
  144. //
  145. #define LEAF_ENTRYCOUNT_THRESHOLD MAX_LEAF_ENTRIES
  146. //#define LEAF_ENTRYCOUNT_THRESHOLD (MAX_LEAF_ENTRIES * 7 / 8)
  147. #define LEAF_SPACE_THRESHOLD 300
  148. #define FRAG_THRESHOLD 400
  149. #define DEF_DEPTH_INCREMENT 2
  150. #define DEF_PAGE_RESERVE 4
  151. #define DEF_PAGE_INCREMENT 32
  152. #define NUM_PAGE_LOCKS 256
  153. #define NUM_TOP_DIR_BITS 9
  154. #define MAX_NUM_TOP_DIR_BITS 10
  155. //
  156. // MASK and SIGNATURES
  157. //
  158. #define DELETE_SIGNATURE 0xCCCC
  159. #define OFFSET_FLAG_DELETED 0x8000
  160. #define OFFSET_VALUE_MASK 0x7fff
  161. #define ART_HEAD_SIGNATURE 0xaaaaaaaa
  162. #define HIST_HEAD_SIGNATURE 0xbbbbbbbb
  163. #define XOVER_HEAD_SIGNATURE 0xcccccccc
  164. #define DEF_HEAD_SIGNATURE 0xdefa1234
  165. #define CACHE_INFO_SIGNATURE 0xbeef0205
  166. #define GROUP_LINK_MASK 0x80000000
  167. //
  168. // history map stuff (should be reg settable)
  169. //
  170. #define DEF_EXPIRE_INTERVAL (3 * SEC_PER_WEEK) // 1 week
  171. #define DEF_CRAWLER_WAKEUP_TIME (30) // 30 secs
  172. #define MIN_MAXPAGES_TO_CRAWL (4)
  173. //
  174. //
  175. // what fraction of total pages to crawl. 1/128 means
  176. // we could cover all the pages in 2 hours. This is
  177. // expressed in terms of shifts. 7 right shift is 128
  178. //
  179. #define FRACTION_TO_CRAWL_SHFT 7
  180. //
  181. // Indicates that the space used for this entry has been reclaimed
  182. //
  183. #define ENTRY_DEL_RECLAIMED ((WORD)0xffff)
  184. //
  185. // Get pointer given the offset and base
  186. //
  187. #define GET_ENTRY( _base, _offset ) \
  188. ((PCHAR)(_base) + (_offset))
  189. //
  190. // See if we need to update the stats in the header page
  191. //
  192. #define UPDATE_HEADER_STATS( ) { \
  193. if ( (m_nInsertions + m_nDeletions) >= STAT_FLUSH_THRESHOLD ) { \
  194. FlushHeaderStats( ); \
  195. } \
  196. }
  197. //
  198. // Current hash table valid version numbers
  199. //
  200. // Significant Versions:
  201. // 000 - old table
  202. // 300 - Removed duplicates from the xover table
  203. // 310 - Remove xposting from article table
  204. // - add articleid/groupid coding into xover table
  205. // 320 - Change XOVER table to hold offsets into articles
  206. // 330 - Change hash table to use two tier directory
  207. // 340 - Change Xover Table to remove most data and place in index files !
  208. // 350 - Change the number of top level directories used to 512 !
  209. //
  210. #define HASH_VERSION_NUMBER_MCIS10 0x340
  211. #define HASH_VERSION_NUMBER 0x350
  212. //
  213. // Disable auto alignments
  214. //
  215. #pragma pack(1)
  216. //
  217. // doubly linked list
  218. //
  219. typedef struct _HASH_LIST_ENTRY {
  220. WORD Flink;
  221. WORD Blink;
  222. } HASH_LIST_ENTRY, *PHASH_LIST_ENTRY;
  223. //
  224. // Entry headers - this is the structure of each hash entry header
  225. //
  226. typedef struct _ENTRYHEADER {
  227. //
  228. // hash value
  229. // *** Must be first entry ***
  230. //
  231. DWORD HashValue;
  232. //
  233. // Size of this entry
  234. // *** Must be second entry ***
  235. //
  236. WORD EntrySize;
  237. #ifdef _WIN64
  238. //
  239. // Alignment word. This makes data align-8 instead of align-2, without
  240. // it the compiler would look at the type "BYTE" figure align-1 is good
  241. // enough and then place it right after EntrySize at an align-2 slot. With
  242. // this word we cause the slot to be 8 bytes into the struct and hence align-8.
  243. // Data will hold larger objects and hence must be aligned better.
  244. //
  245. WORD Reserved;
  246. #endif
  247. //
  248. // Member variable for var length data
  249. //
  250. BYTE Data[1] ;
  251. } ENTRYHEADER, *PENTRYHEADER;
  252. //
  253. // headers for deleted entries
  254. //
  255. typedef struct _DELENTRYHEADER {
  256. //
  257. // doubly linked list
  258. //
  259. HASH_LIST_ENTRY Link;
  260. //
  261. // Size of this entry
  262. //
  263. WORD EntrySize;
  264. //
  265. // ???
  266. //
  267. WORD Reserved;
  268. } DELENTRYHEADER, *PDELENTRYHEADER;
  269. //
  270. // Handle for caching entry information
  271. //
  272. typedef struct _HASH_CACHE_INFO {
  273. //
  274. // Signature to verify this structure
  275. //
  276. DWORD Signature;
  277. //
  278. // Cache of hash and msg id length
  279. //
  280. HASH_VALUE HashValue;
  281. DWORD MsgIdLen;
  282. } HASH_CACHE_INFO, *PHASH_CACHE_INFO;
  283. //
  284. // This is the structure for the head page.
  285. //
  286. typedef struct _HASH_RESERVED_PAGE {
  287. //
  288. // Signature to identify the hash file
  289. //
  290. DWORD Signature;
  291. //
  292. // Version number of the hash table
  293. //
  294. DWORD VersionNumber;
  295. //
  296. // Has this file been initialized?
  297. //
  298. BOOL Initialized;
  299. //
  300. // Is this file active
  301. //
  302. BOOL TableActive;
  303. //
  304. // Number of pages that are being used including the reserved page
  305. //
  306. DWORD NumPages;
  307. //
  308. // Max dir depth of the directory
  309. //
  310. DWORD DirDepth;
  311. //
  312. // Statistics
  313. //
  314. DWORD InsertionCount;
  315. DWORD DeletionCount;
  316. DWORD SearchCount;
  317. DWORD PageSplits;
  318. DWORD DirExpansions;
  319. DWORD TableExpansions;
  320. DWORD DupInserts;
  321. } HASH_RESERVED_PAGE, *PHASH_RESERVED_PAGE;
  322. //
  323. // This is the structure of each entry in the leaf pages of the hash table.
  324. //
  325. typedef struct _MAP_PAGE {
  326. //
  327. // Prefix of hash values currently mapped into this page
  328. //
  329. DWORD HashPrefix;
  330. //
  331. // Number of bits of the hash value that is mapped to this page
  332. //
  333. BYTE PageDepth;
  334. //
  335. // Number of active indices in the page (includes deletes)
  336. //
  337. BYTE EntryCount;
  338. //
  339. // Number of real entries in this page
  340. //
  341. BYTE ActualCount;
  342. //
  343. // Flags
  344. //
  345. BYTE Flags;
  346. //
  347. // Location to add new entry
  348. //
  349. WORD ReservedWord;
  350. WORD NextFree;
  351. //
  352. // Location of the page ending
  353. //
  354. WORD LastFree;
  355. //
  356. // Number of bytes left by deleted entries
  357. //
  358. WORD FragmentedBytes;
  359. //
  360. // Reserved, can be used for anything. !QWA
  361. //
  362. WORD Reserved1;
  363. WORD Reserved2;
  364. WORD Reserved3;
  365. WORD Reserved4;
  366. //
  367. // List of deleted entries
  368. //
  369. HASH_LIST_ENTRY DeleteList;
  370. //
  371. // Offset of entry from beginning of page. Deleted entries
  372. // will have the high bit set.
  373. //
  374. WORD Offset[MAX_LEAF_ENTRIES];
  375. //
  376. // Start of Entries
  377. //
  378. DWORD StartEntries;
  379. } MAP_PAGE, *PMAP_PAGE;
  380. //
  381. // Page header flags
  382. //
  383. #define PAGE_FLAG_SPLIT_IN_PROGRESS (WORD)0x0001
  384. //
  385. // XOVER FLAGS
  386. //
  387. #define XOVER_MAP_PRIMARY ((BYTE)0x01)
  388. #pragma pack()
  389. class CDirectory;
  390. class PageEntry;
  391. class CPageCache ;
  392. typedef CRefPtr< CPageCache > CCACHEPTR ;
  393. //
  394. // Helper class for managing all of the locks that must be manipulated
  395. // to access something within the Hash Tables.
  396. //
  397. // There are two principal locks that must almost always be held :
  398. //
  399. // A Lock on the directory which references a hash table page
  400. // (this can be either exclusive or shared)
  401. // A CDirectory object contains and manages this lock.
  402. //
  403. // A Lock on the page which contains the hash table page -
  404. // A PageEntry structure contains and manages this lock.
  405. //
  406. // What we do is co-ordinate the use of these two locks, whenever
  407. // hash table data is accessed we keep track of both the Directory
  408. // that refers to the hash page and the hash table page containing the
  409. // data. (through pointers to these objects). When we are created
  410. // we always have NULL pointers, and we accumulate these pointers
  411. // as CHashMap
  412. //
  413. //
  414. class CPageLock {
  415. private:
  416. friend class CPageCache ;
  417. friend class CDirectory ;
  418. friend class CHashMap ;
  419. //
  420. // Copy constructor is private - because we don't want people making copies !!
  421. //
  422. CPageLock( CPageLock& ) ;
  423. CPageLock& operator=( CPageLock& ) ;
  424. //
  425. // The PageEntry object which has read our page into memory, and which
  426. // has the critical section guaranteeing exclusive access to the page
  427. //
  428. PageEntry* m_pPage ;
  429. //
  430. // The directory we used to lookup the page - we have grabbed
  431. // the directory either exclusively, or shared - The caller must
  432. // keep track of which and use the appropriate API's
  433. //
  434. CDirectory* m_pDirectory ;
  435. //
  436. // Used to figure out whether we got the page shared or exclusive !
  437. //
  438. BOOL m_fPageShared ;
  439. //
  440. // Secondary page - used when we are doing page splits !
  441. //
  442. PageEntry* m_pPageSecondary ;
  443. #ifdef DEBUG
  444. //
  445. // This is used in debug builds to make sure the caller is using
  446. // the correct API's and keeping track of shared/exclusive usage
  447. // correctly !
  448. //
  449. BOOL m_fExclusive ;
  450. #endif
  451. //
  452. // Only public members are constructor and destructor -
  453. // everything else is used by our friends - CHashMap only !
  454. //
  455. public :
  456. //
  457. // Initialize to a NULL state
  458. //
  459. inline CPageLock() ;
  460. #ifdef DEBUG
  461. //
  462. // In debug builds - _ASSERT check that all our members
  463. // are set to NULL before we are destroyed - as we release locks
  464. // we will set these to NULL !
  465. //
  466. inline ~CPageLock() ;
  467. #endif
  468. private :
  469. //
  470. // Lock a specific directory for shared access, and
  471. // remember what directory we grabbed so we can release it later !
  472. //
  473. inline void AcquireDirectoryShared( CDirectory* pDirectory ) ;
  474. //
  475. // Lock a specific directory for EXCLUSIVE access, and remember what
  476. // directory we grabbed so we can release it later !
  477. //
  478. inline void AcquireDirectoryExclusive( CDirectory* pDirectory ) ;
  479. //
  480. // Release the directory !
  481. //
  482. inline void ReleaseDirectoryExclusive() ;
  483. //
  484. // Release a ShareLock from the directory !
  485. //
  486. public: void ReleaseDirectoryShared() ;
  487. //
  488. // Lock a page - and remember who we locked for later release !
  489. //
  490. private: inline PMAP_PAGE AcquirePageShared(
  491. PageEntry *page,
  492. HANDLE hFile,
  493. DWORD PageNumber,
  494. BOOL fDropDirectory
  495. ) ;
  496. inline PMAP_PAGE AcquirePageExclusive(
  497. PageEntry *page,
  498. HANDLE hFile,
  499. DWORD PageNumber,
  500. BOOL fDropDirectory
  501. ) ;
  502. //
  503. // Lock the secondary page Exclusive !
  504. //
  505. inline BOOL AddPageExclusive(
  506. PageEntry* page,
  507. HANDLE hFile,
  508. DWORD PageNumber
  509. ) ;
  510. //
  511. // Release all of our locks - we were holding a shared lock before !
  512. //
  513. inline void ReleaseAllShared( PMAP_PAGE page ) ;
  514. //
  515. // Release all of our locks - we had an exclusive lock on the directory !
  516. //
  517. inline void ReleaseAllExclusive( PMAP_PAGE page ) ;
  518. //
  519. // Check that all of the members are legally setup !
  520. //
  521. BOOL IsValid() ;
  522. } ;
  523. typedef CPageLock HPAGELOCK ;
  524. extern DWORD LeafMask[];
  525. //
  526. // Size of our directory view
  527. //
  528. #define DIR_VIEW_SIZE (64 * KB)
  529. //
  530. // Number of entries per view
  531. //
  532. #define DIR_VIEW_ENTRIES (16 * KB)
  533. #define DIR_VIEW_SHIFT 14
  534. #define DIR_VIEW_MASK 0x00003fff
  535. //
  536. // Number of inserts and deletes before stats are flushed
  537. //
  538. #define STAT_FLUSH_THRESHOLD 16
  539. DWORD
  540. WINAPI
  541. CrawlerThread(
  542. LPVOID Context
  543. );
  544. //
  545. // This is a protototype for the function pointer the hash tables
  546. // can call in case of critical errors.
  547. // This function is called when the hash table gets an error from
  548. // the O.S. which will seriously affect the usablility of the hash table.
  549. // For instance, if we fail to allocate more memory when expanding a
  550. // directory, the hash table will most likely to start to fail the
  551. // majority of attempts to insert new items.
  552. // This is probably non recoverable, and the function would be called
  553. // with fRecoverable set to FALSE.
  554. // If the hash table fails due to a more temporary condition, ie. out
  555. // of disk space, then the function will be called with fRecoverable
  556. // set to TRUE.
  557. // This function pointer is registered on the call to the hash tables
  558. // Initialize() function.
  559. //
  560. typedef void (* HASH_FAILURE_PFN)( LPVOID lpv,
  561. BOOL fRecoverable ) ;
  562. //
  563. // This interface defines the interface used by both Key's and Entry's
  564. // to save and restore their data into the hash table.
  565. //
  566. class ISerialize {
  567. public :
  568. //
  569. // Save the data into a serialized form, and return a pointer
  570. // to where the next block of data can be placed !
  571. // This can be called IFF IsDataSet() == TRUE
  572. //
  573. virtual LPBYTE Serialize( LPBYTE pbPtr ) const = 0 ;
  574. //
  575. // Restore data from a serialized block of data !
  576. // This can be called IF IsDataSet() == TRUE || IsDataSet() == FALSE
  577. //
  578. // If the return value is NULL the call failed, and cbOut contains
  579. // the number of bytes required to hold the data in unserialized form !
  580. //
  581. virtual LPBYTE Restore( LPBYTE pbPtr,
  582. DWORD &cbOut
  583. ) = 0 ;
  584. //
  585. // Return the number of bytes required to serialize the object !
  586. // This can be called IFF IsDataSet() == TRUE
  587. //
  588. virtual DWORD Size( ) const = 0 ;
  589. //
  590. // This function verifies that the serialized block of data is valid !
  591. // This can be called IF IsDataSet() == TRUE || IsDataSet() == FALSE
  592. //
  593. // NOTE : pbContainer - will point to the start of the entire block
  594. // of data containing both serialized Key and Data blocks !
  595. //
  596. virtual BOOL Verify( LPBYTE pbContainer,
  597. LPBYTE pbPtr,
  598. DWORD cb
  599. ) const = 0 ;
  600. } ;
  601. //
  602. // This interface defines the extra interface members a Key must support
  603. // in order to be used by the hash tables
  604. //
  605. class IKeyInterface : public ISerialize {
  606. protected :
  607. //
  608. // Default parameter for EntryData() when we wish to discard
  609. // the size output !
  610. //
  611. static DWORD cbJunk ;
  612. public:
  613. //
  614. // Compute the hash value of this key !
  615. // this must be called IFF IsDataSet() == TRUE !
  616. //
  617. virtual DWORD Hash() const = 0 ;
  618. //
  619. // Compare a Key to a Serialized Key this must
  620. // be called IFF IsDataSet() == TRUE
  621. //
  622. virtual BOOL CompareKeys( LPBYTE pbPtr ) const = 0 ;
  623. //
  624. // Given some serialized Key Data find the Start of
  625. // the serialized Entry Data. The resulting pointer
  626. // must be suitable to be passed to an Entry object's
  627. // Restore() method.
  628. //
  629. virtual LPBYTE EntryData( LPBYTE pbPtr,
  630. DWORD& cbKeyOut = IKeyInterface::cbJunk
  631. ) const = 0 ;
  632. } ;
  633. class IStringKey : public IKeyInterface {
  634. private :
  635. //
  636. // Pointer to the String We are concerned with !
  637. //
  638. LPBYTE m_lpbKey ;
  639. //
  640. // Number of bytes available !
  641. //
  642. DWORD m_cbKey ;
  643. //
  644. // The structure we serialize to.
  645. //
  646. struct SerializedString {
  647. WORD cb ;
  648. BYTE Data[1] ;
  649. } ;
  650. typedef SerializedString* PDATA ;
  651. public :
  652. //
  653. // Construct a Key object !
  654. //
  655. IStringKey( LPBYTE pbKey = 0,
  656. DWORD cbKey = 0
  657. ) ;
  658. //
  659. // Required by ISerialize Interface - save key data into memory
  660. //
  661. LPBYTE
  662. Serialize( LPBYTE pbPtr ) const ;
  663. //
  664. // Required by ISerialize Interface - restore key data from buffer
  665. //
  666. LPBYTE
  667. Restore( LPBYTE pbPtr, DWORD &cbOut ) ;
  668. //
  669. // Return the number of bytes required by Serialize()
  670. //
  671. DWORD
  672. Size() const ;
  673. //
  674. // Check that a serialized version of the data looks legal !
  675. //
  676. BOOL
  677. Verify( LPBYTE pbContainer,
  678. LPBYTE pbPtr,
  679. DWORD cb
  680. ) const ;
  681. //
  682. // Compute the hash value of the key
  683. //
  684. DWORD
  685. Hash( ) const ;
  686. //
  687. // Compare the Key contained in an instance to a serialized one !
  688. //
  689. BOOL
  690. CompareKeys( LPBYTE pbPtr ) const ;
  691. //
  692. // Get a pointer to where the Entry should have been serialized
  693. // following the Key
  694. //
  695. LPBYTE
  696. EntryData( LPBYTE pbPtr, DWORD &cbOut ) const;
  697. } ;
  698. //
  699. // Interface to be used to access the reserved words within the hash table
  700. //
  701. class IEnumInterface {
  702. public :
  703. //
  704. // This interface is passed to the GetFirstMapEntry, GetNextMapEntry routines !
  705. // This function is called to determine if there is anything on this
  706. // page we wish to deal with !!
  707. //
  708. virtual BOOL
  709. ExaminePage( PMAP_PAGE page ) = 0 ;
  710. //
  711. // This function is called to determine if we wish to examine an
  712. // a particular entry within a page !!
  713. //
  714. //
  715. virtual BOOL
  716. ExamineEntry( PMAP_PAGE page, LPBYTE pbPtr ) = 0 ;
  717. } ;
  718. //
  719. // an entry in the hash table
  720. //
  721. // this is an interface used for writing to data in the hashmaps.
  722. // to store data in the hashmap a class must derive from this that
  723. // implements all of the methods
  724. //
  725. // NOTE : This interface handles the serialization of keys and puts them
  726. // automagically as the first element of the structure. This is laid out
  727. // as specified by _KEY_ENTRY below !
  728. //
  729. typedef struct _KEY_ENTRY {
  730. // length of key
  731. WORD KeyLen;
  732. BYTE Key[1];
  733. } KEY_ENTRY, *PKEY_ENTRY;
  734. //
  735. // CHashEntry is an old interface used by the hash tables.
  736. // we define a
  737. //
  738. //
  739. //
  740. class CHashEntry : public ISerialize
  741. {
  742. public:
  743. LPBYTE
  744. Serialize( LPBYTE pbPtr ) const {
  745. SerializeToPointer( pbPtr ) ;
  746. return pbPtr ;
  747. }
  748. LPBYTE
  749. Restore( LPBYTE pbPtr, DWORD& cbOut ) {
  750. cbOut = 0 ;
  751. RestoreFromPointer( pbPtr ) ;
  752. return pbPtr ;
  753. }
  754. DWORD
  755. Size( ) const {
  756. return GetEntrySize() ;
  757. }
  758. BOOL
  759. Verify( LPBYTE pbContainer, LPBYTE pbPtr, DWORD cbData ) const {
  760. return Verify( pbContainer, (DWORD)(pbPtr-pbContainer), pbPtr) ;
  761. }
  762. //
  763. // take data and store it into a pointer. no more then
  764. // CHashEntry::GetEntrySize() bytes should be stored here
  765. //
  766. virtual void SerializeToPointer(LPBYTE pbPtr) const=0;
  767. //
  768. // unserialize data at pbPtr into this class
  769. //
  770. virtual void RestoreFromPointer(LPBYTE pbPtr)=0;
  771. //
  772. // the number of bytes required to store the data in this
  773. // hash entry
  774. //
  775. virtual DWORD GetEntrySize() const=0;
  776. //
  777. // make sure that data is valid in its marshalled format
  778. //
  779. virtual BOOL Verify(LPBYTE pKey, DWORD cKey, LPBYTE pbPtr) const = 0;
  780. };
  781. class CHashWalkContext
  782. {
  783. private:
  784. //
  785. // variables needed for GetFirstMapEntry/GetNextMapEntry
  786. //
  787. // buffer containing the data in the current page
  788. BYTE m_pPageBuf[HASH_PAGE_SIZE];
  789. // the current page that we are examining
  790. DWORD m_iCurrentPage;
  791. // the current entry in the current page
  792. DWORD m_iPageEntry;
  793. friend class CHashMap;
  794. };
  795. //
  796. // Flags returned from VerifyHashMapFile in *pdwErrorFlags. anything that fits
  797. // in the mask 0x0000ffff is a fatal error (so the hashmap is invalid)
  798. //
  799. // invalid directory link
  800. #define HASH_FLAG_BAD_LINK 0x00000001
  801. // invalid hash file signature
  802. #define HASH_FLAG_BAD_SIGNATURE 0x00000002
  803. // invalid hash file size
  804. #define HASH_FLAG_BAD_SIZE 0x00000004
  805. // corrupt page prefix
  806. #define HASH_FLAG_PAGE_PREFIX_CORRUPT 0x00000008
  807. // file init flag not set
  808. #define HASH_FLAG_NOT_INIT 0x00000010
  809. // page entry count doesn't match number of page entries
  810. #define HASH_FLAG_BAD_ENTRY_COUNT 0x00000020
  811. // invalid page count
  812. #define HASH_FLAG_BAD_PAGE_COUNT 0x00000040
  813. // bad directory depth
  814. #define HASH_FLAG_BAD_DIR_DEPTH 0x00000080
  815. // invalid entry size or offset
  816. #define HASH_FLAG_ENTRY_BAD_SIZE 0x00000100
  817. // invalid entry hash value
  818. #define HASH_FLAG_ENTRY_BAD_HASH 0x00000200
  819. // Verify() function on entry data failed
  820. #define HASH_FLAG_ENTRY_BAD_DATA 0x00000400
  821. //
  822. // hash file couldn't be found
  823. //
  824. #define HASH_FLAG_NO_FILE 0x00010000
  825. //
  826. // If this is set, then no rebuilding is to take place
  827. // because of a fatal error.
  828. //
  829. #define HASH_FLAG_ABORT_SCAN 0x00020000
  830. //
  831. // flags that can be passed into the verify functions to specify how
  832. // strict they should be (more flags means longer checks and slower
  833. // runs).
  834. //
  835. //
  836. // build a directory and check it for integrity. this is also done in
  837. // Initialize, so it is not needed in general
  838. #define HASH_VFLAG_FILE_CHECK_DIRECTORY 0x00000001
  839. // do basic checks for for valid offsets, page lengths, and hash values
  840. // for each entry (this needs to be there for anything to get checked in
  841. // pages)
  842. #define HASH_VFLAG_PAGE_BASIC_CHECKS 0x00010000
  843. // call the CHashEntry::Verify method to check the data of each entry
  844. #define HASH_VFLAG_PAGE_VERIFY_DATA 0x00020000
  845. // check for overlapping entries
  846. #define HASH_VFLAG_PAGE_CHECK_OVERLAP 0x00040000
  847. #define HASH_VFLAG_ALL 0xffffffff
  848. //
  849. // These flags indicate that the file is corrupt and should
  850. // be rebuilt.
  851. //
  852. #define HASH_FLAGS_CORRUPT 0x0000ffff
  853. //
  854. //
  855. //
  856. // CHashMap - pure virtual base class for NNTP hash table
  857. //
  858. // The algorithm we are using is similar to the Extendible Hashing
  859. // Technique specified in "Extendible Hashing - A Fast Access Method for
  860. // Dynamic Files" ACM Transaction on Database, 1979 by Fagin,
  861. // Nievergelt, et.al. This algorithm guarantees at most 2 page faults
  862. // to locate a given key.
  863. //
  864. class CHashMap {
  865. public:
  866. CHashMap();
  867. virtual ~CHashMap( );
  868. //
  869. // verify that a hash file isn't corrupted (fsck/chkdsk for hashmap
  870. // files). this should be called before init
  871. //
  872. static BOOL VerifyHashFile(
  873. LPCSTR HashFileName,
  874. DWORD Signature,
  875. DWORD dwCheckFlags,
  876. DWORD *pdwErrorFlags,
  877. IKeyInterface* pIKey,
  878. ISerialize *pHashEntry);
  879. //
  880. // Initialize the hash table but specify what cache to use !
  881. //
  882. BOOL Initialize(
  883. IN LPCSTR HashFileName,
  884. IN DWORD Signature,
  885. IN DWORD MinimumFileSize,
  886. IN DWORD Fraction,
  887. IN CCACHEPTR pCache,
  888. IN DWORD dwCheckFlags = HASH_VFLAG_PAGE_BASIC_CHECKS,
  889. IN HASH_FAILURE_PFN HashFailurePfn = 0,
  890. IN LPVOID lpvFailureCallback = 0,
  891. IN BOOL fNoBuffering = FALSE
  892. );
  893. //
  894. // Initialize the hash table
  895. // this needs to be called before the hash table is used.
  896. //
  897. BOOL Initialize(
  898. IN LPCSTR HashFileName,
  899. IN DWORD Signature,
  900. IN DWORD MinimumFileSize,
  901. IN DWORD cPageEntry = 256,
  902. IN DWORD cNumLocks = 64,
  903. IN DWORD dwCheckFlags = HASH_VFLAG_PAGE_BASIC_CHECKS,
  904. IN HASH_FAILURE_PFN HashFailurePfn = 0,
  905. IN LPVOID lpvFailureCallback = 0,
  906. IN BOOL fNoBuffering = FALSE
  907. );
  908. //
  909. // Lookup an entry
  910. //
  911. // helper functions for some of the non-obvious uses of this are below
  912. //
  913. BOOL LookupMapEntry( const IKeyInterface* pIKey,
  914. ISerialize* pHashEntry,
  915. BOOL bDelete = FALSE,
  916. BOOL fDirtyOnly = FALSE
  917. );
  918. //
  919. // Delete an entry
  920. //
  921. BOOL DeleteMapEntry( const IKeyInterface* pIKey,
  922. BOOL fDirtyOnly = FALSE ) {
  923. return LookupMapEntry(pIKey, 0, TRUE, fDirtyOnly);
  924. }
  925. //
  926. // Lookup and Delete and entry in one step
  927. //
  928. BOOL LookupAndDelete( const IKeyInterface* pIKey,
  929. ISerialize *pHashEntry) {
  930. return LookupMapEntry(pIKey, pHashEntry, TRUE);
  931. }
  932. //
  933. // See if the entry is here
  934. //
  935. BOOL Contains( const IKeyInterface* pIKey ) {
  936. return LookupMapEntry(pIKey, 0, FALSE);
  937. }
  938. //
  939. // Insert or update a map entry
  940. //
  941. BOOL InsertOrUpdateMapEntry(const IKeyInterface* pIKey,
  942. const ISerialize* pHashEntry,
  943. BOOL bUpdate = FALSE,
  944. BOOL fDirtyOnly = FALSE);
  945. //
  946. // Insert new entry
  947. //
  948. BOOL InsertMapEntry(const IKeyInterface* pIKey,
  949. const ISerialize* pHashEntry,
  950. BOOL fDirtyOnly = FALSE) {
  951. return InsertOrUpdateMapEntry(pIKey, pHashEntry, FALSE, fDirtyOnly);
  952. }
  953. //
  954. // Update Map Entry
  955. //
  956. BOOL UpdateMapEntry(const IKeyInterface* pIKey,
  957. const ISerialize* pHashEntry) {
  958. return InsertOrUpdateMapEntry(pIKey, pHashEntry, TRUE);
  959. }
  960. //
  961. // returns the current number of entries in the hash table
  962. //
  963. DWORD GetEntryCount() const { return(m_nEntries); }
  964. //
  965. // see if the hash table is active
  966. //
  967. inline BOOL IsActive() { return m_active; }
  968. //
  969. // methods for walking the entries in the hashtable. order should be
  970. // considered random.
  971. //
  972. BOOL
  973. GetFirstMapEntry( IKeyInterface* pIKey,
  974. DWORD& cbKeyRequried,
  975. ISerialize* pHashEntry,
  976. DWORD& cbEntryRequried,
  977. CHashWalkContext *pHashWalkContext,
  978. IEnumInterface* pEnum
  979. );
  980. //
  981. // Get the next entry
  982. //
  983. BOOL
  984. GetNextMapEntry( IKeyInterface* pIKey,
  985. DWORD& cbKeyRequried,
  986. ISerialize* pHashEntry,
  987. DWORD& cbEntryRequried,
  988. CHashWalkContext *pHashWalkContext,
  989. IEnumInterface *pEnum
  990. );
  991. //
  992. // Get the next entry in the next page
  993. //
  994. BOOL
  995. GetNextPageEntry( IKeyInterface* pIKey,
  996. DWORD& cbKeyRequried,
  997. ISerialize* pHashEntry,
  998. DWORD& cbEntryRequried,
  999. CHashWalkContext *pHashWalkContext,
  1000. IEnumInterface *pEnum
  1001. );
  1002. //
  1003. // make a backup copy of the hashmap suitable
  1004. //
  1005. BOOL MakeBackup(LPCSTR pszBackupFilename);
  1006. //
  1007. // CRCHash function (which is the default, but can be overridden by
  1008. // overriding the Hash method below)
  1009. //
  1010. static DWORD CRCHash(IN const BYTE * Key, IN DWORD KeyLength);
  1011. static void CRCInit(void);
  1012. protected :
  1013. //
  1014. // Close the hash table
  1015. //
  1016. virtual VOID Shutdown(BOOL fLocksHeld = FALSE);
  1017. private:
  1018. //
  1019. // Create the initial set of directory objects !!!
  1020. //
  1021. DWORD
  1022. InitializeDirectories(
  1023. WORD cBitDepth
  1024. ) ;
  1025. //
  1026. // verify that the page structure is valid
  1027. //
  1028. static BOOL VerifyPage( PMAP_PAGE Page,
  1029. DWORD dwCheckFile,
  1030. DWORD *pdwErrorFlags,
  1031. IKeyInterface* pIKey,
  1032. ISerialize *pHashEntry
  1033. );
  1034. //
  1035. // Allocates and initialize the directory
  1036. //
  1037. DWORD I_BuildDirectory( BOOL SetupHashFile = TRUE );
  1038. //
  1039. // Cleans up the mapping
  1040. //
  1041. VOID I_DestroyPageMapping( VOID );
  1042. //
  1043. // Additional work that needs to be done by the derived class
  1044. // for an entry during an insertion
  1045. //
  1046. virtual VOID I_DoAuxInsertEntry(
  1047. IN PMAP_PAGE MapPage,
  1048. IN DWORD EntryOffset
  1049. ) {}
  1050. //
  1051. // Additional work that needs to be done by the derived class
  1052. // for an entry during a delete
  1053. //
  1054. virtual VOID I_DoAuxDeleteEntry(
  1055. IN PMAP_PAGE MapPage,
  1056. IN DWORD EntryOffset
  1057. ) {}
  1058. //
  1059. // Additional work that needs to be done by the derived class
  1060. // for an entry during a page split
  1061. //
  1062. virtual VOID I_DoAuxPageSplit(
  1063. IN PMAP_PAGE OldPage,
  1064. IN PMAP_PAGE NewPage,
  1065. IN PVOID NewEntry
  1066. ) {}
  1067. //
  1068. // Find next available slot for an entry
  1069. //
  1070. DWORD I_FindNextAvail(
  1071. IN HASH_VALUE HashValue,
  1072. IN PMAP_PAGE MapPage
  1073. );
  1074. //
  1075. // Initializes a brand new hash file
  1076. //
  1077. DWORD I_InitializeHashFile( VOID );
  1078. //
  1079. // Initialize a new leaf
  1080. //
  1081. VOID I_InitializePage(
  1082. IN PMAP_PAGE MapPage,
  1083. IN DWORD HashPrefix,
  1084. IN DWORD PageDepth
  1085. );
  1086. //
  1087. // link the deleted entry to the delete list
  1088. //
  1089. VOID I_LinkDeletedEntry(
  1090. IN PMAP_PAGE MapPage,
  1091. IN DWORD Offset
  1092. );
  1093. //
  1094. // When re-loading a hash table, we need to increase our directory
  1095. // depth on the fly without grabbing any locks etc...
  1096. //
  1097. BOOL I_SetDirectoryDepthAndPointers(
  1098. IN PMAP_PAGE MapPage,
  1099. IN DWORD PageNum
  1100. ) ;
  1101. //
  1102. // Set up links for the given page
  1103. //
  1104. BOOL I_SetDirectoryPointers(
  1105. IN HPAGELOCK& hLock,
  1106. IN PMAP_PAGE MapPage,
  1107. IN DWORD PageNumber,
  1108. IN DWORD MaxDirEntries
  1109. );
  1110. //
  1111. // Open the hash file and set up file mappings
  1112. //
  1113. DWORD I_SetupHashFile( IN BOOL &NewTable );
  1114. //
  1115. // Find a page we can use within the hash table.
  1116. //
  1117. DWORD I_AllocatePageInFile(WORD Depth) ;
  1118. //
  1119. // loads/unloads the correct page view
  1120. //
  1121. inline
  1122. PDWORD LoadDirectoryPointerShared(
  1123. DWORD HashValue,
  1124. HPAGELOCK& hLock
  1125. );
  1126. inline
  1127. PDWORD LoadDirectoryPointerExclusive(
  1128. DWORD HashValue,
  1129. HPAGELOCK& hLock
  1130. );
  1131. //
  1132. // Compare if reserved pages are same
  1133. //
  1134. BOOL CompareReservedPage( HASH_RESERVED_PAGE* page1, HASH_RESERVED_PAGE* page2 );
  1135. //
  1136. // Current depth of directory
  1137. //
  1138. WORD m_dirDepth;
  1139. //
  1140. // Number of bits we use to select a CDirectory object !
  1141. //
  1142. WORD m_TopDirDepth ;
  1143. //
  1144. // Initial depth of leaves
  1145. //
  1146. WORD m_initialPageDepth;
  1147. //
  1148. // Maximum number of entries per leaf page before we split
  1149. //
  1150. WORD m_pageEntryThreshold;
  1151. //
  1152. // Maximum number of bytes used before we split
  1153. //
  1154. WORD m_pageMemThreshold;
  1155. //
  1156. // Name of the hash file
  1157. //
  1158. CHAR m_hashFileName[MAX_PATH];
  1159. //
  1160. // handle to the hash file
  1161. //
  1162. HANDLE m_hFile;
  1163. //
  1164. // Do we want to let NT do buffering of this file !
  1165. //
  1166. BOOL m_fNoBuffering ;
  1167. //
  1168. // pages allocated
  1169. //
  1170. DWORD m_maxPages;
  1171. //
  1172. // Long that we use to synchronize FlushHeaderStats() call -
  1173. // whoever InterlockExchange's and gets a 0 back should go
  1174. // ahead and assume they have the lock !
  1175. //
  1176. long m_UpdateLock ;
  1177. //
  1178. // Critical section for managing allocation of new pages
  1179. //
  1180. CRITICAL_SECTION m_PageAllocator ;
  1181. //
  1182. // Boolean to indicate whether we've had a critical
  1183. // failure in allocating memory, and will be unable
  1184. // to recover simply !
  1185. // ie. If we fail a VirtualAlloc() call, then we set this to
  1186. // TRUE, as we will probably not recover. However, if we have
  1187. // run out of Disk Space, we leave this as FALSE as we will
  1188. // probably be able to recover.
  1189. //
  1190. BOOL m_fCriticalFailure ;
  1191. //
  1192. // A pointer of a function to call in case of severe failures
  1193. // which will crimp the hash tables functionality.
  1194. //
  1195. HASH_FAILURE_PFN m_HashFailurePfn ;
  1196. //
  1197. // An opaque LPVOID we will pass to the m_HashFailurePfn when
  1198. // we call it.
  1199. //
  1200. LPVOID m_lpvHashFailureCallback ;
  1201. //
  1202. // handle to file mapping
  1203. //
  1204. HANDLE m_hFileMapping;
  1205. //
  1206. // pointer to the reserved page
  1207. //
  1208. PHASH_RESERVED_PAGE m_headPage;
  1209. #if 0
  1210. //
  1211. // pointer to the hash file
  1212. //
  1213. //PMAP_PAGE m_hashPages;
  1214. LPVOID m_lpvBuffers ;
  1215. #endif
  1216. //
  1217. // Handle to the directory mapping
  1218. //
  1219. HANDLE m_hDirMap;
  1220. //
  1221. // Locks the directory
  1222. //
  1223. _RWLOCK *m_dirLock;
  1224. //
  1225. // Top level directory
  1226. //
  1227. // DWORD *m_pTopDirectory ;
  1228. //
  1229. // Array of CDirectory objects - used to find out what hash
  1230. // table pages have the data we want !
  1231. //
  1232. CDirectory *m_pDirectory[(1<< MAX_NUM_TOP_DIR_BITS)] ;
  1233. //
  1234. // Pointer to the Cache which holds all of our pages
  1235. //
  1236. CCACHEPTR m_pPageCache ;
  1237. //
  1238. // A power of 2 fraction which indicates what
  1239. // purportion of the cache pages we can occupy !
  1240. //
  1241. DWORD m_Fraction ;
  1242. //
  1243. // Head page signature
  1244. //
  1245. DWORD m_HeadPageSignature;
  1246. //
  1247. // whether the hash table is active or not
  1248. //
  1249. BOOL m_active;
  1250. //
  1251. // methods for GetFirstMapEntry/GetNextMapEntry
  1252. //
  1253. BOOL LoadWalkPage(CHashWalkContext *pHashWalkContext);
  1254. //
  1255. // the flags to pass into VerifyPage when loading pages (0 disables
  1256. // calling VerifyPage)
  1257. //
  1258. DWORD m_dwPageCheckFlags;
  1259. //
  1260. // Set to TRUE if we successfully returned from Initialize(). If this
  1261. // occurs then on shutdown we should be able to save the Directory
  1262. // structure
  1263. //
  1264. BOOL m_fCleanInitialize;
  1265. protected:
  1266. //
  1267. // Hash function
  1268. //
  1269. virtual DWORD Hash(IN LPBYTE Key, IN DWORD KeyLength);
  1270. //
  1271. // acquire directory lock
  1272. //
  1273. VOID AcquireBackupLockShared( );
  1274. VOID AcquireBackupLockExclusive( );
  1275. //
  1276. // acquires the lock for the given directory entry index
  1277. // returns a handle to the actual lock
  1278. //
  1279. PMAP_PAGE AcquireLockSetShared(
  1280. IN DWORD DirEntry,
  1281. OUT HPAGELOCK& lock,
  1282. BOOL fDropDirectory = FALSE
  1283. );
  1284. PMAP_PAGE AcquireLockSetExclusive(
  1285. IN DWORD DirEntry,
  1286. OUT HPAGELOCK& lock,
  1287. BOOL fDropDirectory = FALSE
  1288. );
  1289. //
  1290. // Add a secondary page to the pagelock !
  1291. //
  1292. BOOL AddLockSetExclusive(
  1293. IN DWORD DirEntry,
  1294. OUT HPAGELOCK& lock
  1295. );
  1296. BOOL
  1297. AddPageExclusive(
  1298. IN DWORD PageNum,
  1299. OUT HPAGELOCK& hLock
  1300. ) ;
  1301. //
  1302. // Get the page addresss,
  1303. // Get a shared lock on the directory AND
  1304. // an exclusive lock on the page !
  1305. //
  1306. PMAP_PAGE GetPageExclusive(
  1307. IN HASH_VALUE HashValue,
  1308. OUT HPAGELOCK& hLock
  1309. ) ;
  1310. //
  1311. // Get the page address and also get a shared lock
  1312. // on both the directory and page objects !
  1313. //
  1314. PMAP_PAGE GetDirAndPageShared(
  1315. IN HASH_VALUE HashValue,
  1316. OUT HPAGELOCK& hLock
  1317. );
  1318. //
  1319. // Get the page address, an exclusive lock on the page
  1320. // and an exclusive lock on the directory !
  1321. //
  1322. PMAP_PAGE GetDirAndPageExclusive(
  1323. IN HASH_VALUE HashValue,
  1324. OUT HPAGELOCK& hLock
  1325. );
  1326. //
  1327. // Get the page address and also lock the directory and page
  1328. // given the page number
  1329. //
  1330. PMAP_PAGE GetAndLockPageByNumber(
  1331. IN DWORD PageNumber,
  1332. OUT HPAGELOCK& hLock
  1333. );
  1334. //
  1335. // Get the page address, and do not lock the directory, but
  1336. // do lock the page. Caller must have directory lock already !
  1337. //
  1338. PMAP_PAGE GetAndLockPageByNumberNoDirLock(
  1339. IN DWORD PageNumber,
  1340. OUT HPAGELOCK& hLock
  1341. ) ;
  1342. //
  1343. // See if the next bit is one
  1344. //
  1345. BOOL I_NextBitIsOne( IN HASH_VALUE HashValue, IN DWORD PageDepth ) {
  1346. return (BOOL)(HashValue & LeafMask[PageDepth]);
  1347. }
  1348. //
  1349. // Releases the directory lock
  1350. //
  1351. VOID ReleaseBackupLockShared( );
  1352. VOID ReleaseBackupLockExclusive();
  1353. //
  1354. // releases the lock
  1355. //
  1356. #if 0
  1357. VOID ReleaseLock( PMAP_PAGE page, HPAGELOCK& hLock ) {
  1358. hLock.ReleasePage( page ) ;
  1359. }
  1360. #endif
  1361. //
  1362. // releases both the page lock and the backup lock
  1363. //
  1364. inline VOID
  1365. ReleasePageShared(
  1366. PMAP_PAGE page,
  1367. HPAGELOCK& hLock
  1368. ) ;
  1369. inline VOID
  1370. ReleasePageExclusive(
  1371. PMAP_PAGE page,
  1372. HPAGELOCK& hLock
  1373. ) ;
  1374. //
  1375. // Page compaction
  1376. //
  1377. BOOL CompactPage(
  1378. IN HPAGELOCK& HLock,
  1379. PMAP_PAGE Page
  1380. );
  1381. //
  1382. // Expand hash file
  1383. //
  1384. BOOL ExpandHashFile(
  1385. DWORD NumPagesToAdd = DEF_PAGE_INCREMENT
  1386. );
  1387. //
  1388. // Expands the directory. Directory will grow by
  1389. // a multiple of 2**nBitsExpand.
  1390. //
  1391. BOOL ExpandDirectory(
  1392. HPAGELOCK& hPageLock,
  1393. WORD nBitsExpand = DEF_DEPTH_INCREMENT
  1394. );
  1395. //
  1396. // Find an existing entry
  1397. //
  1398. BOOL FindMapEntry(
  1399. IN const IKeyInterface* pIKey,
  1400. IN HASH_VALUE HashValue,
  1401. IN PMAP_PAGE MapPage,
  1402. IN const ISerialize* pIEntryInterface,
  1403. OUT PDWORD AvailIndex OPTIONAL,
  1404. OUT PDWORD MatchedIndex OPTIONAL
  1405. );
  1406. //
  1407. // Updates the header statistics
  1408. //
  1409. VOID FlushHeaderStats(
  1410. BOOL fLockHeld = FALSE
  1411. );
  1412. //
  1413. // flush a page
  1414. //
  1415. BOOL FlushPage(
  1416. HPAGELOCK& hLock,
  1417. PVOID Base,
  1418. BOOL fDirtyOnly = FALSE
  1419. );
  1420. //
  1421. // Get the remaining bytes in the system
  1422. //
  1423. DWORD GetBytesAvailable( PMAP_PAGE MapPage ) {
  1424. return((DWORD)(MapPage->LastFree - MapPage->NextFree));
  1425. }
  1426. //
  1427. // Given the hash value, get the index to the directory
  1428. // NOTE: m_dirDepth must never be == 0 or else this computation
  1429. // will not work. It is initialized to 2 in the hash table so
  1430. // can never be 0 by design.
  1431. //
  1432. DWORD GetDirIndex( HASH_VALUE HashValue ) {
  1433. return (HashValue >> (32 - m_dirDepth ));
  1434. }
  1435. //
  1436. // Get the leaf entry index
  1437. //
  1438. DWORD GetLeafEntryIndex( HASH_VALUE HashValue ) {
  1439. return (DWORD)(HashValue & LEAF_ENTRY_MASK);
  1440. }
  1441. //
  1442. // Increment global stats
  1443. //
  1444. VOID IncrementInsertCount( VOID ) {
  1445. InterlockedIncrement((PLONG)&m_nInsertions);
  1446. ++m_nEntries;}
  1447. VOID IncrementDeleteCount( VOID ) {
  1448. InterlockedIncrement((PLONG)&m_nDeletions);
  1449. --m_nEntries;}
  1450. VOID IncrementSearchCount( VOID ) { m_nSearches++; }
  1451. VOID IncrementSplitCount( VOID ) { m_nPageSplits++; }
  1452. VOID IncrementDirExpandCount( VOID ) { m_nDirExpansions++; }
  1453. VOID IncrementTableExpandCount( VOID ) { m_nTableExpansions++; }
  1454. VOID IncrementDupInsertCount( VOID ) { m_nDupInserts++; }
  1455. //
  1456. // link the deleted entry to the delete list
  1457. //
  1458. VOID LinkDeletedEntry(
  1459. IN PMAP_PAGE MapPage,
  1460. IN DWORD Offset
  1461. );
  1462. //
  1463. // Allocate a deleted buffer, if possible
  1464. //
  1465. PVOID ReuseDeletedSpace(
  1466. IN PMAP_PAGE MapPage,
  1467. IN HPAGELOCK& HLock,
  1468. IN DWORD & NeededEntrySize
  1469. );
  1470. //
  1471. // Sets a page flag bit
  1472. //
  1473. VOID SetPageFlag(
  1474. PMAP_PAGE MapPage,
  1475. HPAGELOCK& HLock,
  1476. WORD Mask
  1477. ) {
  1478. MapPage->Flags |= (WORD)(Mask);
  1479. FlushPage( HLock, MapPage ) ;
  1480. }
  1481. //
  1482. // Splits a page
  1483. //
  1484. BOOL SplitPage(
  1485. IN PMAP_PAGE Page,
  1486. HPAGELOCK& hLock,
  1487. OUT BOOL & Expand
  1488. );
  1489. //
  1490. // get the size of an entry
  1491. //
  1492. DWORD GetEntrySize( const ISerialize* pIKey,
  1493. const ISerialize* pHashEntry
  1494. );
  1495. //
  1496. // **************************************************************
  1497. // **************************************************************
  1498. //
  1499. // number of Map pages
  1500. //
  1501. DWORD m_nPagesUsed;
  1502. //
  1503. // number of insertions
  1504. //
  1505. DWORD m_nInsertions;
  1506. //
  1507. // number of deletions
  1508. //
  1509. DWORD m_nDeletions;
  1510. //
  1511. // number of entries in the hash table
  1512. //
  1513. DWORD m_nEntries;
  1514. //
  1515. // number of searches
  1516. //
  1517. DWORD m_nSearches;
  1518. //
  1519. // number of duplicate insertions
  1520. //
  1521. DWORD m_nDupInserts;
  1522. //
  1523. // number of pages plits
  1524. //
  1525. DWORD m_nPageSplits;
  1526. //
  1527. // number of times we had to expand the directory
  1528. //
  1529. DWORD m_nDirExpansions;
  1530. //
  1531. // number of times we had to remap the file
  1532. //
  1533. DWORD m_nTableExpansions;
  1534. }; // CHashMap
  1535. //
  1536. // This class defines a hash table which uses LPSTR's as the
  1537. // key !
  1538. //
  1539. class CStringHashMap : private CHashMap {
  1540. public :
  1541. //
  1542. // verify that a hash file isn't corrupted (fsck/chkdsk for hashmap
  1543. // files). this should be called before init
  1544. //
  1545. static BOOL VerifyHashFile(
  1546. LPCSTR HashFileName,
  1547. DWORD Signature,
  1548. DWORD dwCheckFlags,
  1549. DWORD *pdwErrorFlags,
  1550. ISerialize *pHashEntry) {
  1551. IStringKey key ;
  1552. return CHashMap::VerifyHashFile(
  1553. HashFileName,
  1554. Signature,
  1555. dwCheckFlags,
  1556. pdwErrorFlags,
  1557. &key,
  1558. pHashEntry
  1559. ) ;
  1560. }
  1561. //
  1562. // Initialize the hash table
  1563. // this needs to be called before the hash table is used.
  1564. //
  1565. BOOL Initialize(
  1566. IN LPCSTR HashFileName,
  1567. IN DWORD Signature,
  1568. IN DWORD MinimumFileSize,
  1569. IN DWORD cPageEntry = 256,
  1570. IN DWORD cNumLocks = 64,
  1571. IN DWORD dwCheckFlags = HASH_VFLAG_PAGE_BASIC_CHECKS,
  1572. IN HASH_FAILURE_PFN HashFailurePfn = 0,
  1573. IN LPVOID lpvFailureCallback = 0
  1574. ) {
  1575. return CHashMap::Initialize(
  1576. HashFileName,
  1577. Signature,
  1578. MinimumFileSize,
  1579. cPageEntry,
  1580. cNumLocks,
  1581. dwCheckFlags,
  1582. HashFailurePfn,
  1583. lpvFailureCallback
  1584. ) ;
  1585. }
  1586. //
  1587. // Lookup an entry
  1588. //
  1589. // helper functions for some of the non-obvious uses of this are below
  1590. //
  1591. BOOL LookupMapEntry(LPBYTE Key,
  1592. DWORD KeyLen,
  1593. ISerialize *pHashEntry,
  1594. BOOL bDelete = FALSE) {
  1595. IStringKey key( Key, KeyLen ) ;
  1596. return CHashMap::LookupMapEntry( &key, pHashEntry, bDelete ) ;
  1597. }
  1598. //
  1599. // Delete an entry
  1600. //
  1601. BOOL DeleteMapEntry(LPBYTE Key, DWORD KeyLen) {
  1602. return LookupMapEntry(Key, KeyLen, NULL, TRUE);
  1603. }
  1604. //
  1605. // Lookup and Delete and entry in one step
  1606. //
  1607. BOOL LookupAndDelete(LPBYTE Key, DWORD KeyLen, ISerialize *pHashEntry) {
  1608. return LookupMapEntry(Key, KeyLen, pHashEntry, TRUE);
  1609. }
  1610. //
  1611. // See if the entry is here
  1612. //
  1613. BOOL Contains(LPBYTE Key, DWORD KeyLen) {
  1614. return LookupMapEntry(Key, KeyLen, NULL);
  1615. }
  1616. //
  1617. // Insert or update a map entry
  1618. //
  1619. BOOL InsertOrUpdateMapEntry(LPBYTE Key, DWORD KeyLen, const ISerialize *pHashEntry, BOOL bUpdate = FALSE) {
  1620. IStringKey key( Key, KeyLen ) ;
  1621. return CHashMap::InsertOrUpdateMapEntry( &key, pHashEntry, bUpdate ) ;
  1622. }
  1623. //
  1624. // Insert new entry
  1625. //
  1626. BOOL InsertMapEntry(LPBYTE Key, DWORD KeyLen, const ISerialize *pHashEntry) {
  1627. return InsertOrUpdateMapEntry(Key, KeyLen, pHashEntry, FALSE);
  1628. }
  1629. //
  1630. // Update Map Entry
  1631. //
  1632. BOOL UpdateMapEntry(LPBYTE Key, DWORD KeyLen, const ISerialize *pHashEntry) {
  1633. return InsertOrUpdateMapEntry(Key, KeyLen, pHashEntry, TRUE);
  1634. }
  1635. //
  1636. // returns the current number of entries in the hash table
  1637. //
  1638. DWORD GetEntryCount() const { return(m_nEntries); }
  1639. //
  1640. // see if the hash table is active
  1641. //
  1642. inline BOOL IsActive() { return CHashMap::IsActive(); }
  1643. //
  1644. // methods for walking the entries in the hashtable. order should be
  1645. // considered random.
  1646. //
  1647. BOOL GetFirstMapEntry( LPBYTE pKey,
  1648. PDWORD pKeyLen,
  1649. ISerialize *pHashEntry,
  1650. CHashWalkContext *pHashWalkContext) {
  1651. DWORD cbData ;
  1652. IStringKey key( pKey, *pKeyLen ) ;
  1653. BOOL fReturn = CHashMap::GetFirstMapEntry( &key,
  1654. *pKeyLen,
  1655. pHashEntry,
  1656. cbData,
  1657. pHashWalkContext,
  1658. 0
  1659. ) ;
  1660. return fReturn ;
  1661. }
  1662. BOOL GetNextMapEntry( LPBYTE pKey,
  1663. PDWORD pKeyLen,
  1664. ISerialize *pHashEntry,
  1665. CHashWalkContext *pHashWalkContext) {
  1666. DWORD cbData ;
  1667. IStringKey key( pKey, *pKeyLen ) ;
  1668. BOOL fReturn = CHashMap::GetNextMapEntry( &key,
  1669. *pKeyLen,
  1670. pHashEntry,
  1671. cbData,
  1672. pHashWalkContext,
  1673. 0
  1674. ) ;
  1675. return fReturn ;
  1676. }
  1677. //
  1678. // make a backup copy of the hashmap suitable
  1679. //
  1680. BOOL MakeBackup(LPCSTR pszBackupFilename) {
  1681. return CHashMap::MakeBackup( pszBackupFilename ) ;
  1682. }
  1683. } ;
  1684. //
  1685. // This class defines a page cache that can be shared amongst hash tables !
  1686. //
  1687. //
  1688. class CPageCache : public CRefCount {
  1689. private :
  1690. LPVOID m_lpvBuffers ;
  1691. //
  1692. // Number of PageLock objects we use to keep track of our cache.
  1693. //
  1694. DWORD m_cPageEntry ;
  1695. //
  1696. // Pointer to an array of PageEntry objects !
  1697. //
  1698. PageEntry* m_pPageEntry ;
  1699. //
  1700. // Track the number of locks we are using to sync access to the pages
  1701. //
  1702. DWORD m_cpageLock ;
  1703. //
  1704. // Pointer to the array of locks we are using !
  1705. //
  1706. _RWLOCK* m_ppageLock ;
  1707. //
  1708. // No copying of CPageCache objects !
  1709. //
  1710. CPageCache( CPageCache& ) ;
  1711. CPageCache& operator=( CPageCache& ) ;
  1712. public :
  1713. CPageCache() ;
  1714. ~CPageCache() ;
  1715. BOOL
  1716. Initialize( DWORD cPageEntry = 0,
  1717. DWORD cLocks = 0
  1718. ) ;
  1719. inline PMAP_PAGE
  1720. AcquireCachePageShared(
  1721. IN HANDLE hFile,
  1722. IN DWORD PageNumber,
  1723. IN DWORD Fraction,
  1724. OUT HPAGELOCK& lock,
  1725. IN BOOL fDropDirectory
  1726. ) ;
  1727. inline PMAP_PAGE
  1728. AcquireCachePageExclusive(
  1729. IN HANDLE hFile,
  1730. IN DWORD PageNumber,
  1731. IN DWORD Fraction,
  1732. OUT HPAGELOCK& lock,
  1733. BOOL fDropDirectory
  1734. ) ;
  1735. inline BOOL
  1736. AddCachePageExclusive(
  1737. IN HANDLE hFile,
  1738. IN DWORD PageNumber,
  1739. IN DWORD Fraction,
  1740. OUT HPAGELOCK& lock
  1741. ) ;
  1742. //
  1743. // releases both the page lock and the backup lock
  1744. //
  1745. static inline VOID
  1746. ReleasePageShared(
  1747. PMAP_PAGE page,
  1748. HPAGELOCK& hLock
  1749. ) ;
  1750. static inline VOID
  1751. ReleasePageExclusive(
  1752. PMAP_PAGE page,
  1753. HPAGELOCK& hLock
  1754. ) ;
  1755. //
  1756. // Remove this file handle from the cache wherever it appears !
  1757. //
  1758. void
  1759. FlushFileFromCache(
  1760. IN HANDLE hFile
  1761. ) ;
  1762. } ;
  1763. DWORD
  1764. CalcNumPagesPerIO( DWORD nPages );
  1765. #include "hashimp.h"
  1766. #endif