_         _                    _ _     
  _   _  ___  _   _| |_ _   _| |__   ___        __| | |__  
 | | | |/ _ \| | | | __| | | | '_ \ / _ \_____ / _` | '_ \ 
 | |_| | (_) | |_| | |_| |_| | |_) |  __/_____| (_| | |_) |
  \__, |\___/ \__,_|\__|\__,_|_.__/ \___|      \__,_|_.__/ 
  |___/__       _                         _ _   __  
      / /__ _  | | __  __ _     _   _  __| | |__\ \ 
     | |/ _` | | |/ / / _` |   | | | |/ _` | '_ \| |
     | | (_| |_|   < | (_| |_  | |_| | (_| | |_) | |
     | |\__,_(_)_|\_(_)__,_(_)  \__, |\__,_|_.__/| |
      \_\                       |___/           /_/ 

Version: 2014aug05
This file:
http://angg.twu.net/youtube-db/README.html
See:
http://angg.twu.net/ferramentas-para-ativistas.html
http://angg.twu.net/ferramentas-para-ativistas.html#testes-javascript
http://angg.twu.net/ferramentas-para-ativistas.html#testes-audio
http://angg.twu.net/ferramentas-para-ativistas.html#testes-video



What is this
============
This - ydb - is the software that I use to maintain the archive of
videos at:

  http://angg.twu.net/linkdasruas2.html

It can be used as a Lua library and as a toolbox, in arbitrarily
complex ways; that is what I do at home to produce linkdasruas2.html
from a very big database spread into several files. But what matters
here is that ydb can also be used as a standalone script, invoked like
this (details below!):

  ydb -simple myindex

to produce a "myindex.html" from a "myindex.txt", possibly downloading
local copies of the videos that the "myindex.txt" point to.



The main idea
=============
Take a look at "linkdasruas2.html". It is generated from a file
"linkdasruas2.txt", that you can access here:

  http://angg.twu.net/linkdasruas2.txt.html
  http://angg.twu.net/linkdasruas2.txt

The HTML contains in copy of that .txt in a <pre>...</pre> block,
followed by a copy of this ".lst",

  http://angg.twu.net/linkdasruas2.lst.html
  http://angg.twu.net/linkdasruas2.lst

For each line in the .txt with a link to youtube, e.g.,

  http://www.youtube.com/watch?v=QJjOM7SpbhM J.S. Bach 6th Cello Suite, Courante

the .lst contains the name of the "local copy" of that video, which
can be something like:

  videos/J.S._Bach_6th_Cello_Suite_transcribed_for_viola_-_Courante-QJjOM7SpbhM.mp4

Note that the 11-char identifier, "QJjOM7SpbhM", is the same in the
URL and in the MP4 file. All the modules in ydb use these identifiers
to relate URLs to files.




Dependencies
============
You will need lua5.1 and lua-posix. On Debian-based systems you can
install them with just:

  sudo apt-get install lua5.1 lua-posix

You will need a _very recent_ version of youtube-dl for downloading
videos from Youtube. This page has installation instructions:

  http://rg3.github.io/youtube-dl/download.html

You can do lots of tests and stuff without youtube-dl, though.

This is optional but recommended:

  sudo apt-get install tkdiff




Downloading, installing, testing
================================
Do this (the "{}"s are optional, but nice if you are a beginner
pasting blocks of commands to a terminal):

 {
  rm -Rv ~/ydb/
  mkdir  ~/ydb/
  cd     ~/ydb/
  wget http://angg.twu.net/youtube-db/ydb.tgz
  tar -xvzf ydb.tgz
 }

 {
  cp -v ~/ydb/test1.txt /tmp/
  cd /tmp/
  ~/ydb/ydb -fake -simple test1
  source test1.sh
  ~/ydb/ydb -fake -simple test1
  mv test1.txt2 test1.txt
  ~/ydb/ydb -fake -simple test1
 }  

and then open this URL in a browser:

  file:///tmp/test1.html

The output messages of each run of "~/ydb/ydb -simple ..." should be
clear enough.

In the first run, ydb detects which youtube URLs in the .txt don't
have local mp4s, dates or titles; all these things can be downloaded
from youtube with youtube-dl, and ydb produces a shell script,
"test1.sh", that downloads all of them - but as we are using the
"-fake" option ydb creates a script test1.sh that creates empty MP4
files and fake titles and dates instead of calling youtube-dl.

The command ". test1.sh" runs the (fake) shell script, which
"downloads" fake mp4s to /tmp/videos/, and stores fake dates and
titles into /tmp/ydbtmp/.

When we run

  ~/ydb/ydb -fake -simple test1

the second time, it finds the mp4s files in /tmp/videos/ and the
"downloaded" dates and times in /tmp/ydbtmp/. It uses the names of
these mp4s to produce a file test1.lst, and from test1.txt and
test1.lst it produces a test1.html. It also creates a file test1.txt2
by adding the missing dates and titles to test1.txt. Normally we would
compare test1.txt and test1.txt2 visually to be sure that the changes
are ok, but in the script above we pretend that the they are indeed
ok, and we simply run

  mv test1.txt2 test1.txt

to override the old test1.txt with the new version.

When we run

  ~/ydb/ydb -fake -simple test1

for the third time, it generates the .lst and the .html again, but now
we know that the .lst has one .mp4 for each youtube URL in the .txt,
and we also know that the .txt is "complete" in the sense that each
URL line in it has a date and a title, so the .html is "complete" too.
We browse the result, at:

  file:///tmp/test1.html

and that's it.




Cleaning up
===========
Run:

 {
  rm -v  /tmp/test1.*
  rm -Rv /tmp/videos/
  rm -Rv /tmp/ydbtmp/
 }


A test without "-fake"
======================
After cleaning up, do:

 {
  cp -v ~/ydb/test1.txt /tmp/
  cd /tmp/
  ~/ydb/ydb -simple test1
  . test1.sh
  ~/ydb/ydb -simple test1
  mv test1.txt2 test1.txt
  ~/ydb/ydb -simple test1
 }

and browse:

  file:///tmp/test1.html

Now change test1.txt - add URLs of videos that interest you, remove
the ones that don't, etc, and run you-know-what to download local
copies and to generate the .html (and the .txt2)...




Cheers
======
The documentation ends abruptly here because this is the first public
release. =)

  Eduardo Ochs
  eduardoochs@gmail.com
  https://www.facebook.com/eduardo.ochs
  http://angg.twu.net/ferramentas-para-ativistas.html#casos-particulares
  http://angg.twu.net/ferramentas-para-ativistas.html#audios
  http://angg.twu.net/ferramentas-para-ativistas.html#testes-javascript
  http://angg.twu.net/ferramentas-para-ativistas.html#testes-videos