Assuming, you let the agency know you needed time off when you interviewed, a week would be sufficient to get someone else in. There's always the chance that, when you give notice, they will "let you go" on the spot. How has your relationship with the agency been up til now?
Some agencies are extremely vindictive, and others will not consider you if you have been temping for the competition, and the other has nothing more for you. If you leave without notice, it becomes part of your record, so if you are expecting a good job reference, you won't get it.