2012/03/29

SimpleDB - "Too many value tests per predicate in the query expression"

當用Simple下Querey的時候,condition超過20個就會出現下面的錯誤
Too many value tests per predicate in the query expression
或
Client error : Too many value tests per predicate in the query expression.

像這類的語法都不行:
SELECT * FROM domain WHERE id='1' OR id='2' OR ... id='20' OR id='21'

SELECT * FROM domain WHERE id in ('1', '2', ... '20', '21')

SELECT * FROM domain WHERE id in ('1', '2', ... '20') OR id in ('21', ...)

只能考慮分批一次抓20個,然後再合併處理。



2012/05/07 發現一種可以破解20個條件限制的辦法:
SELECT * FROM `domain` WHERE 
  account in ('key1', 'key2', 'key3', 'key4', 'key5', 'key6', 'key7', 'key8', 'key9', 'key10', 
    'key11', 'key12', 'key13', 'key14', 'key15', 'key16', 'key17', 'key18', 'key19', 'key20') OR 
  dummy IS NOT NULL OR 
  account in ('key21', 'key22', 'key23', 'key24', 'key25', 'key26', 'key27', 'key28', 'key29', 'key30', 
    'key31', 'key32', 'key33', 'key34', 'key35', 'key36', 'key37', 'key38', 'key39', 'key40')
沒錯,就是加了"OR dummy IS NOT NULL"這個多餘的條件在中間就可以下達多個條件了!

不過當你有這樣的需求時,就應該想一下是不是設計面有問題,或是該用 relational database了。



參考資料
Amazon SimpleDB » Developer Guide » Amazon SimpleDB Concepts » Limits
Query 101: Building Amazon SimpleDB Queries

2012/03/28

架設 Subversion system + apache

安裝SVN
yum install subversion apache mod_dav_svn

在此我們不用 daemon mode (svnserve),而是使用 apache 來管理

設定帳密
htpasswd -c /etc/svn/passwd user

設定權限
[groups]
admin = user1, user2

[Project1:/]
@admin = rw

[Project2:/]
@admin = rw

[Project2:/Doc]
user3 = rw

設定apache configuration
/etc/httpd/conf.d/subversion.conf
LoadModule dav_svn_module     modules/mod_dav_svn.so
LoadModule authz_svn_module   modules/mod_authz_svn.so

<Location /svn>
  DAV svn
  SVNParentPath /var/svn

  SSLRequireSSL
  AuthType Basic
  AuthName "Authorization Realm"
  AuthUserFile /etc/svn/passwd
  AuthzSVNAccessFile /etc/svn/authz
  Require valid-user
</location>

建立新的 project
svnadmin create /var/svn/Project1

讓 httpd 有存取 svn 目錄的權限
chown -R apache:apache /var/svn/Project1

最後開啓網址 https://localhost/svn/Project1 試試看


2012/03/27

編譯 binary 的時候自動加入 release version 與 date

Makefile
SVN_VERSION = "\"`svn info | grep Revision | sed 's/Revision: //g'`\""
BUILD_TIME = "\"`date '+%Y/%m/%d %H:%M:%S'`\""

CFLAGS += -DSVN_VERSION=$(SVN_VERSION) -DBUILD_TIME=$(BUILD_TIME)

.c 檔
void show_version (void)
{
    printf("SVN Version: %s, Bulid time: %s\n", SVN_VERSION, BUILD_TIME);
}
Preprocessing 的時候 macro 會自動被帶換掉啦!

2012/03/25

用 shell script 做一個假的 DNS

很有趣的問題,突然跟我同事在想要怎麼快速做一個假的DNS (Domain Name Server)? 不就follow 鳥哥的教學,用 bind 架一個就好了?
有想過寫在 /etc/hosts,但發現 nslookup 並不會去查詢 hosts file

Anyway, geek 就喜歡搞怪,喜歡繞道而行,喜歡自己造輪子...

=> 想到用 netcat 當 dns proxy,把 dns query redirect 給 8.8.8.8,但遇到 special 的 domain name 時,就自己 return。



Step1: 測試 dns proxy
$ sudo nc -v -u -l -s 127.0.0.1 -p 53 -e 'nc -u 8.8.8.8 53'
$ nslookup - 127.0.0.1
> yahoo.com
Server:         10.32.100.199
Address:        10.32.100.199#53
Non-authoritative answer:
Name:   yahoo.com
Address: 209.191.122.70
Name:   yahoo.com
Address: 72.30.38.140
Name:   yahoo.com
Address: 98.139.183.24
> 
It's work!

想看整個過程,開三個session
$ mkfifo fifo1 fifo2
$ sudo nc -u -l -s 127.0.0.1 -p 53 -c -x > fifo1 < fifo2
Received 27 bytes from the socket
00000000  E2 20 01 00  00 01 00 00  00 00 00 00  05 79 61 68  . ...........yah
00000010  6F 6F 03 63  6F 6D 00 00  01 00 01                  oo.com.....
Sent 75 bytes to the socket
00000000  E2 20 81 80  00 01 00 03  00 00 00 00  05 79 61 68  . ...........yah
00000010  6F 6F 03 63  6F 6D 00 00  01 00 01 C0  0C 00 01 00  oo.com..........
00000020  01 00 00 09  80 00 04 62  8B B7 18 C0  0C 00 01 00  .......b........
00000030  01 00 00 09  80 00 04 D1  BF 7A 46 C0  0C 00 01 00  .........zF.....
00000040  01 00 00 09  80 00 04 48  1E 26 8C                  .......H.&.

$ nc -u 8.8.8.8 53 -x < fifo1 > fifo2
Sent 27 bytes to the socket
00000000  E2 20 01 00  00 01 00 00  00 00 00 00  05 79 61 68  . ...........yah
00000010  6F 6F 03 63  6F 6D 00 00  01 00 01                  oo.com.....
Received 75 bytes from the socket
00000000  E2 20 81 80  00 01 00 03  00 00 00 00  05 79 61 68  . ...........yah
00000010  6F 6F 03 63  6F 6D 00 00  01 00 01 C0  0C 00 01 00  oo.com..........
00000020  01 00 00 09  80 00 04 62  8B B7 18 C0  0C 00 01 00  .......b........
00000030  01 00 00 09  80 00 04 D1  BF 7A 46 C0  0C 00 01 00  .........zF.....
00000040  01 00 00 09  80 00 04 48  1E 26 8C                  .......H.&.

$ nslookup - 127.0.0.1
> yahoo.com
Server:         10.32.100.199
Address:        10.32.100.199#53

Non-authoritative answer:
Name:   yahoo.com
Address: 98.139.183.24
Name:   yahoo.com
Address: 209.191.122.70
Name:   yahoo.com
Address: 72.30.38.140
>


Step 2: 截取 dns query
$ sudo nc -u -l -s 127.0.0.1 -p 53 > fake_query
$ nslookup - 127.0.0.1
> yahoo.com


Step 3: 用 fake_query 去 get response
$ cat fake_query | nc -u 8.8.8.8 53 > fake_response


Step 4: 把 fake_response 內的 ip 換成特定的 ip
(這一步還沒做)


Step 5: Start fake dns
$ mkfifo fifo1 fifo2
$ sudo nc -u -l -s 127.0.0.1 -p 53 -c > fifo1 < fifo2 &
$ sh filter.sh < fifo1 > fifo2

filter.sh 裡面會截取 current query id,然後 replace fake response 裡面的
#!/bin/sh
QID=
COUNT=0
cat - | while read -n 1 byte
do
    if [ $COUNT -lt 2 ]; then
        QID+=$byte
    fi

    if [ $COUNT -eq 2 ]; then
        echo -n $QID$(cut -b 3- fake_response)
    fi

    COUNT=`expr $COUNT + 1`
done


結果
$ nslookup - 127.0.0.1
> yahoo.com
;; Got bad packet: bad compression pointer
51 bytes
65 9c 81 80 01 03 05 79 61 68 6f 6f 03 63 6f 6d
01 01 c0 0c 01 01 01 ad 04 d1 bf 7a 46 c0 0c 01
01 01 ad 04 48 1e 26 8c c0 0c 01 01 01 ad 04 62
8b b7 18
>

Oh no~ 還是有問題,初估應該是 dns packet header 哪邊需要動態更改...
Alright! 是該停止這浪費時間的行為了...



參考資料
想利用 shell script 做到這些事情代價真大,以下是必需了解的:
  • 以 hex mode 顯示 binary file
hexdump -C file
while read -n 1 byte; do
    ord=$(printf "%b" "${byte:-\000}" |
          od -t x1 |
          { read offset hex; echo $hex; })
    echo "$ord"
done
#!/bin/sh
cat - | while read LINE
do
  echo ${LINE}
done

try ls -al | script.sh

2012/03/21

Google app engine (GAE) - Hello World

Step by step 紀錄我的學習過程:

1. 申請 Google app engine

2. 環境
  • Python 2.5.2 - MacOS X有內建

3. 了解 Google App 限制

4. 了解 Google App 提供的資源

5. 下載SDK Python 專用的 Google 應用服務引擎 SDK
我自己是用 mac 版的圖形介面,按滑鼠右鍵選擇 new,然後選擇自己的 project directory,再來點左上角的 Run,就可以透過 localhost 去預覽自己的 app:
6. IDE - Eclipse
下載 Google Plugin for Eclipse

話說安裝的時候一直遇到下面的錯誤,最後換了一個網路環境就可以了,應該是網路問題吧?
An error occurred while collecting items to be installed
session context was:(profile=epp.package.jee, phase=org.eclipse.equinox.internal.p2.engine.phases.Collect, operand=, action=).
HTTP Server 'Bad Gateway' : http://dl.google.com/eclipse/plugin/3.7/plugins/com.google.gdt.eclipse.gph.subversive_2.5.2.v201202290255-rel-r37.jar
HttpClient connection error response code 502.


7. 範例 - Hello, World!
下載後,記得修改 app.yaml,application name 要填自己的才有辦法 deploy 到 server
application: ysl-test1
version: 1
runtime: python
api_version: 1

handlers:
- url: /.*
  script: helloworld.py


8. 部署 - 使用 appcfg.py
我自己是使用OS X的圖形介面,按一下右上角的 Deploy 就自動部署上到 google 了,部署成功訊息如下:

9. 瀏覽 http://(application_name).appspot.com 看看 Hello World有沒有 show 出來

10. 後台管理
  • 系統監控
  • 檢視 Log
  • Application 設定


小結
Google app engine架設網站相當快速、非常適合懶得自己架 server (IDC or AWS) 的人使用,所有的 application setting 透過 Web UI 就可以達成。
但跟 google 綁這麼緊密,使用它的data storage、API、管理工具、Log... 將來如果想轉到其他環境...應該會很難。



參考資料

2012/03/20

Build OpenCV error

我在編譯 OpenCV-2.3.1 for python 時遇到下面的錯誤

錯誤一:Linking error
$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D BUILD_PYTHON_SUPPORT=ON -D  PYTHON_INCLUDE_DIR=/usr/local/include/python2.6 -D PYTHON_LIBRARY=/usr/local/lib/libpython2.6.a ..
...
/usr/bin/ld: /usr/local/lib/libpython2.7.a(abstract.o): relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/libpython2.7.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make[2]: *** [lib/cv2.so] Error 1
make[1]: *** [modules/python/CMakeFiles/opencv_python.dir/all] Error 2
make: *** [all] Error 2

解決方法就是重新編譯 python 加上 --enable-shared 的參數...找了好久lol
$ ./configure --enable-shared && make && make install


錯誤二:在 python 內找不到 cv module
Python 2.6.7 (r267:88850, Mar 20 2012, 11:54:21) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv
Traceback (most recent call last):
  File "stdin", line 1, in module
ImportError: No module named cv
>>>
/module/stdin

從 InstallGuide 得知解決方法:
1. 把cv.so和cv2.so copy到
2. 增加path到PYTHONPATH
export PYTHONPATH=~/projects/opencv/release/lib:$PYTHONPATH

2012/03/19

工作站帶著走 - shell環境設定

環境設定檔

  • /etc/bashrc
  • /etc/profile
  • ~/.bashrc
  • ~/.profile or ~/.bash_profile

就是login時apply的shell檔,執行的順序可以看鳥哥的教學

/etc/profile
export PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin

~/.bashrc
alias ll='ls -al'
alias n='netstat -ntulp'
alias w='watch -n 1'

export LANG=en
export SVN_EDITOR=vim
export LD_LIBRARY_PATH=/usr/local/lib

有想到什麼陸續再補充

參考資料
鳥哥的 Linux 私房菜 -- 學習 bash shell