-- begin REPLACE SCRAPING WITH API --------------------------------------------------------------------------------------------------------------------- make a github release version in REFERENCE.txt, say which version still has legacy scraping code in tact remove scraping code. don't 'move' it MOVE TO LEGACY -------------- - scraping code - "legacy inAudible wire-up code" Book.IsEpisodes Book.HasBookDetails LibraryBook.DownloadBookLink humanizer replace all scraping with audible api public partial class ScanLibraryDialog : Form, IIndexLibraryDialog public async Task DoMainWorkAsync() using var pageRetriever = websiteProcessorControl1.GetPageRetriever(); jsonFilepaths = await DownloadLibrary.DownloadLibraryAsync(pageRetriever).ConfigureAwait(false); scraping stuff to remove. order matters. go from top to bottom: REMOVED PROJECT REFERENCES 3.2 Domain Utilities (post database) DomainServices AudibleDotComAutomation 3.1 Domain Internal Utilities InternalUtilities Scraping REMOVED PROJECTS 2 Utilities (domain ignorant) Scraping AudibleDotComAutomation AudibleDotCom CookieMonster REMOVED FILES \Libation\InternalUtilities\UNTESTED\DataConverter.cs \Libation\DomainServices\UNTESTED\DownloadLibrary.cs \Libation\DomainServices\UNTESTED\ScrapeBookDetails.cs ADDED PROJECT REFERENCES 3.1 Domain Internal Utilities InternalUtilities DTOs -- end REPLACE SCRAPING WITH API --------------------------------------------------------------------------------------------------------------------- -- begin ENHANCEMENT, PERFORMANCE: IMPORT --------------------------------------------------------------------------------------------------------------------- imports are PAINFULLY slow for just a few hundred items. wtf is taking so long? -- end ENHANCEMENT, PERFORMANCE: IMPORT --------------------------------------------------------------------------------------------------------------------- -- begin ENHANCEMENT, PERFORMANCE: GRID --------------------------------------------------------------------------------------------------------------------- when a book/pdf is NOT liberated, calculating the grid's [Liberated][NOT d/l'ed] label is very slow https://stackoverflow.com/a/12046333 https://codereview.stackexchange.com/a/135074 // do NOT use lock() or Monitor with async/await private static int _lockFlag = 0; // 0 - free if (Interlocked.CompareExchange(ref _lockFlag, 1, 0) != 0) return; // only 1 thread will enter here without locking the object/put the other threads to sleep try { await DoWorkAsync(); } // free the lock finally { Interlocked.Decrement(ref _lockFlag); } use stop light icons for liberated state: red=none, yellow=downloaded encrypted, green=liberated need a way to liberate ad hoc books and pdf.s use pdf icon with and without and X over it to indicate status -- end ENHANCEMENT, PERFORMANCE: GRID --------------------------------------------------------------------------------------------------------------------- -- begin ENHANCEMENT, GET LIBRARY --------------------------------------------------------------------------------------------------------------------- Audible API. GET /1.0/library , GET /1.0/library/{asin} TONS of expensive conversion: GetLibraryAsync > string > JObject > string > LibraryApiV10 -- end ENHANCEMENT, GET LIBRARY --------------------------------------------------------------------------------------------------------------------- -- begin ENHANCEMENT, DEBUGGING --------------------------------------------------------------------------------------------------------------------- datalayer stuff (eg: Book) need better ToString -- end ENHANCEMENT, DEBUGGING --------------------------------------------------------------------------------------------------------------------- -- begin BUG, MOVING FILES --------------------------------------------------------------------------------------------------------------------- with libation closed, move files start libation can get error below fixed on restart Form1_Load ... await setBackupCountsAsync(); Collection was modified; enumeration operation may not execute. stack trace at System.ThrowHelper.ThrowInvalidOperationException_InvalidOperation_EnumFailedVersion() at System.Collections.Generic.List`1.Enumerator.MoveNextRare() at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source, Func`2 predicate) at FileManager.FilePathCache.GetPath(String id, FileType type) in C:\Dropbox\Dinah's folder\coding\_NET\Visual Studio 2019\Libation\FileManager\UNTESTED\FilePathCache.cs:line 33 at FileManager.AudibleFileStorage.d__32.MoveNext() in C:\Dropbox\Dinah's folder\coding\_NET\Visual Studio 2019\Libation\FileManager\UNTESTED\AudibleFileStorage.cs:line 112 at FileManager.AudibleFileStorage.d__31.MoveNext() in C:\Dropbox\Dinah's folder\coding\_NET\Visual Studio 2019\Libation\FileManager\UNTESTED\AudibleFileStorage.cs:line 107 at FileManager.AudibleFileStorage.d__30.MoveNext() in C:\Dropbox\Dinah's folder\coding\_NET\Visual Studio 2019\Libation\FileManager\UNTESTED\AudibleFileStorage.cs:line 104 at LibationWinForm.Form1.<g__getAudioFileStateAsync|15_1>d.MoveNext() in C:\Dropbox\Dinah's folder\coding\_NET\Visual Studio 2019\Libation\LibationWinForm\UNTESTED\Form1.cs:line 110 at LibationWinForm.Form1.d__15.MoveNext() in C:\Dropbox\Dinah's folder\coding\_NET\Visual Studio 2019\Libation\LibationWinForm\UNTESTED\Form1.cs:line 117 at LibationWinForm.Form1.d__13.MoveNext() in C:\Dropbox\Dinah's folder\coding\_NET\Visual Studio 2019\Libation\LibationWinForm\UNTESTED\Form1.cs:line 81 at LibationWinForm.Form1.d__11.MoveNext() in C:\Dropbox\Dinah's folder\coding\_NET\Visual Studio 2019\Libation\LibationWinForm\UNTESTED\Form1.cs:line 60 -- end BUG, MOVING FILES --------------------------------------------------------------------------------------------------------------------- -- begin CONFIG FILES --------------------------------------------------------------------------------------------------------------------- .\appsettings.json should only be a pointer to the real settings file location: LibationSettings.json replace complex config saving throughout with new way in my ConsoleDependencyInjection solution all settings should be strongly typed re-create my shortcuts and bak multiple files named "appsettings.json" will overwrite each other libraries should avoid this generic name. ok for applications to use them Audible API C:\Dropbox\Dinah's folder\coding\_NET\Visual Studio 2019\audible api\AudibleApi\_Tests\AudibleApi.Tests\bin\Debug\netcoreapp3.0\L1 C:\Dropbox\Dinah's folder\coding\_NET\Visual Studio 2019\audible api\AudibleApi\_Tests\AudibleApi.Tests\bin\Debug\netcoreapp3.0\ComputedTestValues 14+ json files these can go in a shared solution folder BasePath => recursively search directories upward-only until fild dir with .sln from here can set up a shared dir anywhere. use recursive upward search to find it. store shared files here. eg: identityTokens.json, ComputedTestValues -- end CONFIG FILES --------------------------------------------------------------------------------------------------------------------- -- begin TAGS --------------------------------------------------------------------------------------------------------------------- pulling previous tags into new Books. think: reloading db move out of Book and into DtoMapper? Extract file and tag stuff from domain objects. This should exist only in data layer. If domain objects are able to call EF context, it should go through data layer Why are tags in file AND database? extract FileManager dependency from data layer -- end TAGS --------------------------------------------------------------------------------------------------------------------- -- begin ENHANCEMENT, CATEGORIES --------------------------------------------------------------------------------------------------------------------- add support for multiple categories when i do this, learn about the different CategoryLadder.Root enums. probably only need Root.Genres -- end ENHANCEMENT, CATEGORIES --------------------------------------------------------------------------------------------------------------------- -- begin CLEAN UP ARCHITECTURE --------------------------------------------------------------------------------------------------------------------- my ui sucks. it's also tightly coupled with biz logic. can't replace ui until biz logic is extracted and loosely. remove all biz logic from presentation/winforms layer -- end CLEAN UP ARCHITECTURE --------------------------------------------------------------------------------------------------------------------- -- begin UNIT TESTS --------------------------------------------------------------------------------------------------------------------- all "UNTESTED" code needs unit tests Turn into unit tests or demos TextBoxBaseTextWriter.cs EnumerationExamples.cs EnumExt.cs IEnumerable[T]Ext.cs MultiTextWriter.cs Selenium.Examples.cs ScratchPad.cs ProcessorAutomationController.Examples.cs ScraperRules.Examples.cs ByFactory.Example.cs search 'example code' on: LibationWinForm\...\Form1.cs EnumerationFlagsExtensions.EXAMPLES() // examples scratchpad scratch pad scratch_pad -- end UNIT TESTS --------------------------------------------------------------------------------------------------------------------- -- begin DECRYPTING --------------------------------------------------------------------------------------------------------------------- replace inaudible/inaudible lite with pure ffmpeg benefits of inaudible: highly configurable embedded cover image chapter-ized cue and nfo files can hopefully get most of this with simple decrypt. possibly including the new chapter titles better chapters in many m4b files. to see, try re-downloading. examples: bobiverse, sharp objects raw ffmpeg decrypting is significantly faster than inAudible/AaxDecrypter method: inAudible/AaxDecrypter tiny file 40 sec decrypt 40 sec chapterize + tag huge file 60 sec decrypt 120 sec chapterize + tag directly call ffmpeg (decrypt only) tiny file 17 sec decrypt huge file 39 sec decrypt -- end DECRYPTING --------------------------------------------------------------------------------------------------------------------- -- begin ENHANCEMENT, UI: LONG RUNNING TASKS --------------------------------------------------------------------------------------------------------------------- long running tasks are appropriately async. however there's no way for the user to see that the task is running (vs nothing happened) except to wait and see if the final notification ever comes need events to update UI with progress -- end ENHANCEMENT, UI: LONG RUNNING TASKS --------------------------------------------------------------------------------------------------------------------- -- begin ENHANCEMENT, PERFORMANCE: TAGS --------------------------------------------------------------------------------------------------------------------- tag edits still take forever and block UI unlikely to be an issue with file write. in fact, should probably roll back this change also touches parts of code which: db write via a hook, search engine re-index -- end ENHANCEMENT, PERFORMANCE: TAGS --------------------------------------------------------------------------------------------------------------------- -- begin ENHANCEMENT: REMOVE BOOK --------------------------------------------------------------------------------------------------------------------- how to remove a book? previously difficult due to implementation details regarding scraping and importing. should be trivial after api replaces scraping -- end ENHANCEMENT: REMOVE BOOK --------------------------------------------------------------------------------------------------------------------- -- begin ENHANCEMENT: NEW VIEWS --------------------------------------------------------------------------------------------------------------------- menu views. filter could work for grid display; just use the lucene query language 1) menu to show all tags and count of each. click on tag so see only those books 2) tree to show all categories and subcategories. click on category so see only those books -- end ENHANCEMENT: NEW VIEWS --------------------------------------------------------------------------------------------------------------------- -- begin ENHANCEMENT: LOGGING, ERROR HANDLING --------------------------------------------------------------------------------------------------------------------- LibationWinForm and Audible API need better logging and error handling incl log levels, db query logging see AaxDecryptorWinForms.initLogging() -- end ENHANCEMENT: LOGGING ---------------------------------------------------------------------------------------------------------------------