|
|
|
|
First, I want to state, for the record, that we received excellent support from John Barton during this eval He was always only a phonecall/email away, and he and I spent many hours of quality time together on the phone and virtual room, looking at problems and solutions together. Johns support was excellent, and so I feel confident that our test results are not skewed by any lack of understanding of the product on my part.. he was there with me every step of the way to keep me straight!
Regarding follow-up, we are currently taking stock of our findings and roadmap options.
Dom Ruffatto is my product management counterpart, and will be the one to contact you regarding any follow-on discussions. I CCed Dom on this email, by way of introduction.
Dom Peter Do is the Sybase account rep for HP.
Regarding testing results.. here is a very quick summary of my findings:
- Select Star Schema queries degraded in performance very quickly as the fact table grew, due to expensive hash joins on queries that matched larger number of dimension records. See chart below. We experimented with join indexes which did speed up the query, but prohibitively slowed down the data loads. John has all the supporting information and query plans
So we modified the schema (and indices) and test queries to focus on a flat fact table structure, where facts and dimensions were pre-joined during loading, with the goal that hash joins could be avoided.
However, data loading times are problematic.. You can see from the chart below that load times increase with the size of the fact table. John helped me profile this behavior.. the elapsed time was almost all involved in the maintenance of two HG indexes applied to high cardinality name columns.. Disk I/O was pegged at 100% (iostats x) during these loads.. This implies that performance cannot be improved by further parallelization.. in fact the addition of additional parallel workloads can be presumed to compound the problem by competing for the same I/O resource.
Removing the HG indices dramatically speeds up the load times, but also dramatically slows down queries which filter on the name columns. LF indices are not an option as the name columns have more than 10,000 unique values.
So, thats the summary of findings.. Please let me know if you feel these findings are in any way inaccurate or unrepresentative of SybaseIQ 15_0 behavior, or if there are any solutions we havent explored relating to the load I/O performance issues mentioned above.
NOTE: We tested Sybase 15.0 beta, understanding that it had improved parallelization of HG index updates, but unfortunately the bottleneck is disk I/O, not CPU.
Also, please let me know if you are aware of any upcoming product features that might improve the load performance of fact tables with HG indexes, specifically enhancements that might reduce the disk I/O associated with these updates and therefore remove or diminish the effects of I/O saturation noted above.
Detailed log:
I. 01/23 18:23:34. 0000025215 [20895]: Insert Pass 1 completed in 5 seconds.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C2_HG: Thread 7 Inserting 98210 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C2_HG: Thread 0 Inserting 98210 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C2_HG: Thread 2 Inserting 98210 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 4 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 0 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 5 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 2 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 3 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 6 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 1 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 7 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C2_HG: Thread 6 Inserting 98210 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C2_HG: Thread 4 Inserting 98210 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 DBA.fact_tmp.ASIQ_IDX_T733_C2_HG Sort Insert Statistics: 30 pages over 1 run(s) (Min: 30 Max: 30 Avg: 30) from 1 inserter(s).
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 4 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 5 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 0 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 2 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 7 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 6 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 3 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 1 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 5 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 4 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 0 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 7 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 6 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 1 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 3 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 2 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 5 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 4 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 0 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 7 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 6 Inserting 16370 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 [20024]: Index DBA.fact_tmp.ASIQ_IDX_T733_C1_HG: Thread 1 Inserting 16320 Recs, 0 Secs.
I. 01/23 18:23:35. 0000025215 DBA.fact_tmp.ASIQ_IDX_T733_C1_HG Sort Insert Statistics: 30 pages over 1 run(s) (Min: 30 Max: 30 Avg: 30) from 1 inserter(s).
I. 01/23 18:23:35. 0000025215 [20895]: Insert Pass 2 completed in 1 seconds.
I. 01/23 18:23:35. 0000025215 [20896]: Insert for 'fact_tmp' completed in 6 seconds. 491050 rows inserted.
I. 01/23 18:23:36. 0000025215 Cmt 85137
I. 01/23 18:23:36. 0000025215 PostCmt 0
I. 01/23 18:23:36. 0000025215 Cmt 85139
I. 01/23 18:23:36. 0000025215 PostCmt 0
I. 01/23 18:23:38. 0000025215 Insert Started.
I. 01/23 18:23:46. 0000025216 Cmt 85142
I. 01/23 18:23:46. 0000025216 PostCmt 0
I. 01/23 18:23:47. 0000025215 [20895]: Insert Pass 1 completed in 11 seconds.
I. 01/23 18:23:50. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 6 Inserting 16380 Recs, 2 Secs.
I. 01/23 18:23:51. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 4 Inserting 16380 Recs, 4 Secs.
I. 01/23 18:23:51. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 2 Inserting 16375 Recs, 4 Secs.
I. 01/23 18:23:52. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 3 Inserting 16380 Recs, 5 Secs.
I. 01/23 18:23:52. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 0 Inserting 16390 Recs, 5 Secs.
I. 01/23 18:24:00. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 5 Inserting 16305 Recs, 13 Secs.
I. 01/23 18:24:05. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 4 Inserting 16380 Recs, 18 Secs.
I. 01/23 18:24:05. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 1 Inserting 16380 Recs, 18 Secs.
I. 01/23 18:24:06. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 6 Inserting 16380 Recs, 19 Secs.
I. 01/23 18:24:06. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 5 Inserting 16380 Recs, 19 Secs.
I. 01/23 18:24:06. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 7 Inserting 16380 Recs, 18 Secs.
I. 01/23 18:24:06. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 2 Inserting 16380 Recs, 19 Secs.
I. 01/23 18:24:06. 0000025217 Cmt 85144
I. 01/23 18:24:06. 0000025217 PostCmt 0
I. 01/23 18:24:07. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 3 Inserting 16380 Recs, 20 Secs.
I. 01/23 18:24:07. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 0 Inserting 16380 Recs, 20 Secs.
I. 01/23 18:24:14. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 3 Inserting 16380 Recs, 22 Secs.
I. 01/23 18:24:17. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 7 Inserting 16360 Recs, 30 Secs.
I. 01/23 18:24:21. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 4 Inserting 16380 Recs, 16 Secs.
I. 01/23 18:24:22. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 6 Inserting 16380 Recs, 16 Secs.
I. 01/23 18:24:22. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 1 Inserting 16380 Recs, 16 Secs.
I. 01/23 18:24:23. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 5 Inserting 16380 Recs, 17 Secs.
I. 01/23 18:24:25. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 2 Inserting 16380 Recs, 19 Secs.
I. 01/23 18:24:25. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 3 Inserting 16360 Recs, 11 Secs.
I. 01/23 18:24:25. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 1 Inserting 16375 Recs, 38 Secs.
I. 01/23 18:24:25. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 7 Inserting 16380 Recs, 19 Secs.
I. 01/23 18:24:26. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 5 Inserting 16390 Recs, 26 Secs.
I. 01/23 18:24:26. 0000025218 Cmt 85146
I. 01/23 18:24:26. 0000025218 PostCmt 0
I. 01/23 18:24:26. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 7 Inserting 16315 Recs, 9 Secs.
I. 01/23 18:24:29. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 3 Inserting 16380 Recs, 22 Secs.
I. 01/23 18:24:29. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 0 Inserting 16380 Recs, 22 Secs.
I. 01/23 18:24:29. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 3 Inserting 16435 Recs, 4 Secs.
I. 01/23 18:24:37. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 1 Inserting 16340 Recs, 11 Secs.
I. 01/23 18:24:37. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 2 Inserting 16380 Recs, 46 Secs.
I. 01/23 18:24:37. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 5 Inserting 16435 Recs, 11 Secs.
I. 01/23 18:24:43. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 6 Inserting 16475 Recs, 53 Secs.
I. 01/23 18:24:46. 0000025219 Cmt 85148
I. 01/23 18:24:46. 0000025219 PostCmt 0
I. 01/23 18:24:47. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 4 Inserting 16380 Recs, 26 Secs.
I. 01/23 18:24:48. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 6 Inserting 16380 Recs, 26 Secs.
I. 01/23 18:24:48. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 1 Inserting 16380 Recs, 26 Secs.
I. 01/23 18:24:51. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 2 Inserting 16380 Recs, 26 Secs.
I. 01/23 18:24:51. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 5 Inserting 16380 Recs, 27 Secs.
I. 01/23 18:24:53. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 7 Inserting 16380 Recs, 27 Secs.
I. 01/23 18:24:53. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 6 Inserting 16380 Recs, 10 Secs.
I. 01/23 18:24:57. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 0 Inserting 16380 Recs, 28 Secs.
I. 01/23 18:24:58. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 3 Inserting 16380 Recs, 29 Secs.
I. 01/23 18:24:59. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 3 Inserting 16350 Recs, 30 Secs.
I. 01/23 18:25:02. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 0 Inserting 16380 Recs, 70 Secs.
I. 01/23 18:25:03. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 6 Inserting 16375 Recs, 9 Secs.
I. 01/23 18:25:04. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 0 Inserting 16350 Recs, 2 Secs.
I. 01/23 18:25:04. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 6 Inserting 16400 Recs, 1 Secs.
I. 01/23 18:25:06. 0000025220 Cmt 85150
I. 01/23 18:25:06. 0000025220 PostCmt 0
I. 01/23 18:25:08. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 3 Inserting 16375 Recs, 9 Secs.
I. 01/23 18:25:09. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 0 Inserting 16050 Recs, 5 Secs.
I. 01/23 18:25:12. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 4 Inserting 16380 Recs, 24 Secs.
I. 01/23 18:25:13. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 7 Inserting 16395 Recs, 47 Secs.
I. 01/23 18:25:13. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 6 Inserting 16380 Recs, 25 Secs.
I. 01/23 18:25:13. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 2 Inserting 16380 Recs, 22 Secs.
I. 01/23 18:25:14. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 1 Inserting 16380 Recs, 26 Secs.
I. 01/23 18:25:14. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 5 Inserting 16380 Recs, 23 Secs.
I. 01/23 18:25:14. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 4 Inserting 16370 Recs, 83 Secs.
I. 01/23 18:25:14. 0000025215 [20024]: Index DBA.fact.fact_dim_config_qualifiedifname_HG: Thread 7 Inserting 16030 Recs, 21 Secs.
I. 01/23 18:25:14. 0000025215 DBA.fact.fact_dim_config_qualifiedifname_HG Sort Insert Statistics: 120 pages over 1 run(s) (Min: 120 Max: 120 Avg: 120) from 1 inserter(s).
I. 01/23 18:25:15. 0000025215 [20024]: Index DBA.fact.fact_dim_time_datetime_min_HG: Thread 1 Inserting 98210 Recs, 0 Secs.
I. 01/23 18:25:15. 0000025215 [20024]: Index DBA.fact.fact_dim_time_datetime_min_HG: Thread 4 Inserting 98210 Recs, 0 Secs.
I. 01/23 18:25:15. 0000025215 [20024]: Index DBA.fact.fact_dim_time_datetime_min_HG: Thread 3 Inserting 98210 Recs, 0 Secs.
I. 01/23 18:25:15. 0000025215 [20024]: Index DBA.fact.fact_dim_time_datetime_min_HG: Thread 5 Inserting 98210 Recs, 0 Secs.
I. 01/23 18:25:15. 0000025215 [20024]: Index DBA.fact.fact_dim_time_datetime_min_HG: Thread 2 Inserting 98210 Recs, 0 Secs.
I. 01/23 18:25:15. 0000025215 DBA.fact.fact_dim_time_datetime_min_HG Sort Insert Statistics: 30 pages over 1 run(s) (Min: 30 Max: 30 Avg: 30) from 1 inserter(s).
I. 01/23 18:25:18. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 2 Inserting 16380 Recs, 41 Secs.
I. 01/23 18:25:19. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 5 Inserting 16390 Recs, 41 Secs.
I. 01/23 18:25:26. 0000025221 Cmt 85152
I. 01/23 18:25:26. 0000025221 PostCmt 0
I. 01/23 18:25:32. 0000025215 [20024]: Index DBA.fact.fact_dim_config_dev_name_HG: Thread 1 Inserting 16400 Recs, 55 Secs.
I. 01/23 18:25:32. 0000025215 DBA.fact.fact_dim_config_dev_name_HG Sort Insert Statistics: 90 pages over 1 run(s) (Min: 90 Max: 90 Avg: 90) from 1 inserter(s).
I. 01/23 18:25:32. 0000025215 [20895]: Insert Pass 2 completed in 105 seconds.
I. 01/23 18:25:32. 0000025215 [20896]: Insert for 'fact' completed in 114 seconds. 491050 rows inserted.
I. 01/23 18:25:40. 0000025215 Cmt 85153
I. 01/23 18:25:40. 0000025215 PostCmt 0
I. 01/23 18:25:41. 0000025215 Cmt 85155
I. 01/23 18:25:41. 0000025215 PostCmt 0
I. 01/23 18:25:41. 0000025215 Cmt 85157
I. 01/23 18:25:41. 0000025215 PostCmt 0
I. 01/23 18:25:41. 0000025215 Cmt 85159
I. 01/23 18:25:41. 0000025215 PostCmt 0
I. 01/23 18:25:46. 0000025222 Cmt 85161
I. 01/23 18:25:46. 0000025222 PostCmt 0 |
|
|
|